The chief technology officer of one of the world’s largest hedge funds talks data.
There’s a weird way in which a hedge fund is a confluence of everything. There’s the money of course—Two Sigma, located in lower Manhattan, manages over $50 billion, an amount that has grown 600 percent in 6 years and is roughly the size of the economy of Bulgaria. Then there are the people—financiers, philosophers, engineers—all applying themselves to unearthing inscrutable patterns that separate fortune from failure....MUCH MORE
And there is the science and engineering, much of it resting on a towering stack of data. In principle, almost any information about the real world can be relevant to a hedge fund. Employees, so the stories go, have camped out next to harbors noting down tanker waterlines, and in retail parking lots counting cars. This data then has to be standardized, synthesized, and made accessible to the people who place bets on the market.
Building the tools to do this is part of Alfred Spector’s job. As the chief technology officer of Two Sigma, he is responsible for the engineering platforms used by the firm’s modelers. A former vice president at IBM and former professor of computer science at Carnegie Mellon, Spector has seen software transform one industry after another, and made more than a few contributions of his own along the way.
He sat down with us for a conversation at Two Sigma’s headquarters earlier this month.
Why would a company like Two Sigma run a public game competition?
We started the Halite AI Programming Competition because we want to be known in the tech community for doing things that the tech community likes. Game competitions are one of those things. Programmers like interesting programing challenges—particularly ones that are contained enough that they can do off-hours. Plus we open-source everything so that the programmers can actually see the game environment and learn everything about the game. That gives programmers a lot of opportunities for creativity as to how the game can be played, and it becomes more fun.
What effect did you notice the competition having on the Two Sigma brand?
What we noticed is that when we go to campuses, people have heard of us more. We also hired someone that was at the top of the ranking.
What was the goal of the game?
Both this year and last year’s games are turn-based strategy games. In Halite 1 last year there were between two and six bots on the board to start. In the game, each bot starts with a single piece, which it can move up, down, left, or right. It can also stay still, in which case the piece gains strength. If it moves, it leaves behind another piece where it was, so the number of pieces on your side grows. All the players are doing this on a grid that’s maybe 40 or 50 squared pieces, so a bot can make a huge number of different moves. This year’s game, Halite 2, is in many ways similar, but it uses a space war theme, where a ship can move to a planet and take over the planet. When you put all this together, people come up with incredibly interesting winning strategies.
What’s an example of an interesting strategy that came out of the games?
There was an interesting, unanticipated strategy that emerged in the final week of last year’s game. It was a non-aggression strategy. Some players determined that if they hung loose for a while, and didn’t try to defeat other players, and just tried to gain space and stay out of trouble, it could help them. Remaining aggressive players would actually sort of hurt themselves, while the players that were being non-aggressive would actually then be in a position that, with moderately high probability, could win.
How was machine learning relevant to game strategy?
