There. It's very difficult to get there, and it shifts the focus from creating value and revenue to looking good and calming down lay people who now control you, plus feeding an army of parasites charging a bajillion.
Today, it's also prone to sudden random fluctuations because an algo-trading software Bayes classifier decided to sell based on historical data it pulled out of its shiny metal ass.
I've said this a few times but we're going through a growth period like AAA video games have over the past 20 years.
I used to be that 2 guys could make a video game, then it went to 10, then 50, now its around 200 from what I've last heard.
Hedge funds are going through a similar shift.
It used to be that one person could manage data cleaning, and algo generation for a fund.
Then cleaning got split out into its own job.
Then the number of data streams exploded growing by a couple orders of magnitude.
Then the data types diverged so that each new data stream needs its own special cleaning, and normalization and even data storage, ie some data isn't suitable for a sql or non sql database storage, like satellite images.
Nowadays a typical algo fund might make use of 100 different algos for trading, each of which has 20 different inputs, some real time, some updated irregularly.
It takes those signals and weights them to come up with a trading signal, which then gets mixed with a portfolio balancing signals and risk signals.
It can be tough to disentangle each individual signal from the algos themselves so even things like detecting if a signal still has alpha generating abilities is tough.
You can have 10 people just back testing signals and monitoring risk levels.
And the growth of data and data sources isn't slowing down.
This is good if you are one of the larger players, see Virtu buying out competitor KCG, who previously ate competitor Knight Capital, yes that fund with the huge blowup, but not so great news if you want to remain a small, person wise, fund.
Not sure how to run a quant fund anymore with only 4 people. Not sure anything an be done about.
Framing it as machine learning undersells the problem.
It's a hybrid model trading in an adversarial, real-dollar environment. The leverage comes from having a small human team trade big volume, much more than they could possibly trade directly, by augmenting their human abilities with automation and a model. Or seen from the other side, it's a model with human oversight.
Any system like that is high risk, high reward. All the successful ones started out by losing a lot of money. Paypal lost an incredible amount to fraud before they started breaking even. OpenDoor lost an incredible amount to mispricing, and took on a ton of balance sheet risk, before their business really started working.
It's automation taking away the lower level trader and analyst positions. These were jobs that were very lucrative. This creates an even more rarefied, winner-take-all atmosphere, and makes an industry that was already considered amoral downright sociopathic.
It looks like after-the-fact trend change detection and then the AI just predicts a target price. Not completely useless, but trading requires a bit more modeling and also a lot of devops to ensure you don't get stuck in a trade if the computer crashes.
We are going to see more cases like this, as market-making robots increase their trade volumes. Big players are pouring more and more money into automated trading, and that trading relies on algorithms. Often the algorithms involve some sort of machine learning, and therefore depend on the input data, which is vulnerable to manipulation.
For example, imagine a bot programmed by some 5-10 person team at a hedge fund. They find that running sentiment analysis on twitter and news comments can accurately predict whether a given security will rise or fall in response to a fed press release. The profits are good, so the team manager moves a couple million into the algorithm. One night after work, a team member tells his programmer friend about the algorithm, and mentions some of the most profitable stocks. This friend goes home and programs another bot to corrupt the sentiment analysis dataset, by posting fake comments with properly tuned sentiment. The hedge fund bot reacts as expected, and now the friend has the power to manipulate the bot. He has outsmarted the bot and can take advantage of the high volume trading.
That might be a bit of a contrived example, but corruption in machine learning data is a very real problem. People are just starting to study it. [0]
One of the things about financial markets is that large numbers of people are attempting to spot patterns, and eek out a profit from the patterns repeating. People are very good at this - which means that, over time, the number of profitable patterns reduces. Thus, to human eyes, market behaviour becomes noise : just like zip compression reduces a bytestream to being essentially white noise by taking out repetitive sequences.
Computers are just the next step, crunching out the patterns until they are unexploitable (below the threshold of trading costs).
The end result is that markets are a random walk - unless you are at the bleeding edge with faster machines, better latency, lower transaction costs, etc.
Of course, an alternative to this is to do true bottom-up analysis, or invest in illiquid companies (like VCs do).
I'd argue that there's more good than bad about algorithmic trading. People making decisions based on fear and adrenaline is much more dangerous than setting a pre-determined course and sticking to a mathematical model. Besides, it's not like they just set up these programs and forget about them. If there's some sort of flaw in the algorithm the trader isn't just going to bang his head into the wall while he loses millions; he's going to fix the algorithm.
It's worse than even that. Because top-tier hedge funds and investment banks can afford the genuinely insane fees to put their gear as physically close to the exchange as possible they can effectively front-run everyone else including other algo systems, which in situations like this it is particularly advantageous. The nature of the inequal playing field, combined with warring algos mean there is now no real connection whatsoever between an instrument's price and reality below a certain time period. Insane doesn't begin to cover what we've created and the power it wields.
I'm happy for Quantoplan making steps to improve the user experience around algorithmic trading, however armchair traders and many hedge funds have been using platforms like Tradestation (and dozens of others) to back-test and develop algorithmic strategy for well over a decade.
I spent a year working as a researcher for a now-failed hedge fund (failed due to regulatory issues, not performance, we were doing 20% year over year on commodity futures). As an engineer/math guy, it was an incredibly interesting experience because it opened my mind to all sorts of theoretical possibilities and explorations of pattern matching, noise filtering, exotic concepts like wavelets and more.
However, I quickly learned a few things from experienced traders and from seeing my work move from testing to prod. What I learned makes me extremely hesitant to employ automated trading systems on my own money:
1. Historical back-testing is a great way to curve fit. You can hyper optimize your algorithm looking for arbitrage opportunities, trends, whatever. You'll get performance reports that make it seem like you're ready to print money. Then you get out and trade and discover that your system can't keep up with market movements because the indicators that you relied upon may have exhibited correlation but not causation.
2. The boon of algorithmic trading is that it attempts to remove emotion from the trading process, not that it is a better predictor. Listening to a machine should help alleviate the symptoms of "fear and greed" that lead to abrupt, incorrect decision making. Think about that for a minute, some hedge funds advocate algorithms not because of predictive power but as guarantees of rational decision making.
3. Conversely, while developing and testing a system, a smart person will almost inevitably try to bring in exotic concepts into price prediction, order sizing and trend following functions. Given enough time, complexity will increase until it becomes challenging to understand the rationale behind a system's output. Trading this way is scary because real money is being moved without an understanding of fundamental and macro-factors.
4. You will very likely lose money. Even at the size of our fund (1B under management) we were sometimes at the mercy of market makers who gave us crap prices on trades or seemingly manipulated prices to hit our stop orders and cause us to exit positions too early.
I love seeing the ideas behind algorithmic trading popularized, however I want to make sure that anyone embarking on it understands the market as a system and not just as a time series to be modelled. It's composed of real human beings, with emotions running wild. If you decide to play, then play, but do so wisely and carefully and remember to keep it simple.
What I mean is that now computers algorithms are making the trades, it is so fucked up now that people pay hundreds of millions of dollars to be physically near the stock exchange so that they have less lag.
I should have been more clear, computer based autonomous or semi-autonomous algorithms should not be allowed to trade.
Ah, but those quants are going to make a bunch of money...
There's a cottage industry forming around finding market distortions caused by bad algos. You'd think that there wouldn't be a bunch of bots running around making stupid decisions, but there are a lot of bots that haven't been updated in some time and were put in place according to some idealized rule-based model in some esoteric area of finance that one guy came up with.
It's kind of like asking yourself seriously how many computers out there are running outdated software. The less fundamentals matter, the more computers handle various activities that have a more or less direct impact on prices of marketable securities, the more this is a valid strategy. It's basically anomaly detection on very messy data.
If transaction costs keep coming down and data becomes cheaper (both of which seem likely), trading as an independent might become less ludicrous than it is. And to be clear, it is currently ludicrous. Transaction costs will eat up any potential gains on the retail side unless you're operating with very large amounts of money.
As a veteran of one algo shop, I have this to say:
Play with the data all you like. Don't try to trade on it if you don't really know what you're doing. (Or, just recklessly trade other people's money. It's fun.)
What you're seeing here is the "napsterization of finance." (Google it, it will lead you to the article I am almost plagiarizing).
Basically, the market at large puts together a pot of money (called "alpha", debatably) The better you are at trading, the more of that pot you get.
BUT this is not a zero sum game. It's worse.
If the markets are functioning properly, then the better you are a this, the bigger the share of the pot you get, AND the smaller the pot of money gets.
It used to be that middlemen like the NYSE stock market specialists made very large amounts of money doing what Homer Simpson automated with a drinky bird. Now, the also shops have already shrunk that pot considerably. Good news for your pension fund. Bad news for you if you try this yourself. So don't.
Like 25 years ago I was writing software for financial traders back when trading was mostly done in open-outcry pits. What we had was very skilled human experts, so our systems were all basically supportive. Some larger rival, famous for their technology, thought they could do better by giving traders Newton-esque handheld computers that would receive new information via infrared networking. Then you didn't need smart traders, just monkeys who would do what the computer told them. Everybody was very impressed with all this high-tech gadgetry.
But one day we hit a period with high volatility, lots of price crashing and spiking. The handheld systems and the servers that drove them were too laggy to keep up, so the monkeys would get taken advantage of both on the way up and on the way down. It was a very expensive lesson for the execs who put too much reliance on the tech and not enough on the expert humans.
It seems so wild to me that not only are people making the same mistake, but at a much, much larger scale. Half a billion dollars lost. So far!
There are a few problems with turning your laptop into a money machine using data analysis.
Remember the maxim, past performance is not a guarantee of future results. You can develop strategies based on past data that will beat the market, but, the nature of markets is to adapt to kill your edge. Markets adapt constantly and your edge stops working at an unknown point in time. It's unknowable when that WILL happen because past data can't show that.
The other reason is transaction costs. In gambling called vig. Let's say I'm betting NFL games. NFL home teams win 51% of games. Even flipping a coin I've read come up heads 50.1% of the time. These are profitable systems. But you're paying the bookie 10% on each loss. You could find someone to bet you on coin tosses and bet heads each time. You have a positive expected return, although you need a huge number of flips to make money!
In trading of course costs is commissions. Why do you think there was a rise in HFT? The strategies are consistently profitable. (Besides the flashing/manipulation tactics) It is ONLY profitable because of extremely low commission costs that are not available to the retail (or even semi-professional) trader.
Systems that can pull $0.0001 out of every share traded overall on high volume can be (pretty easily) created, but you can't trade them profitably. In fact, you will find commissions (semi-pros who pay about $3 per 1000 shares) priced right at the point of an edge you could be expected to develop.
The conclusion of the article: Automated Trading could be dangerous to the market as such and should be closely monitored. They appear disturbingly similar to the hedge funds of the old days. Besides, quant-finance big shots like consider automated trading based on algorithms risky.
Doesn't this almost guarantee profit, as long as the average price doesn't come down too much? As in, buy when it's [low enough] and sell once it's [high enough]? Of course, if the market is filled with these algorithms it will get harder, but low-risk investments and trading over long periods of time should turn to notable profit, no?
Weird stuff, machine pushing the man out of business.
Today, it's also prone to sudden random fluctuations because an algo-trading software Bayes classifier decided to sell based on historical data it pulled out of its shiny metal ass.
reply