I thought the biggest issue wasn't with the models themselves, but how Zillow decided to apply and act on them, which is why it didn't work in practice. So on average their predictions may have been pretty good, but since each transaction also depends on the other party to accept their offer, and whether they get outbid, most of their predictions where the offer actually goes through would be on the tail end of where they slightly overestimated the price. This tweet from the article summed it up nicely > Zillow made the same mistake that every new quant trader makes early on: Mistaking an adversarial environment for a random one. twitter.com/0xdoug/status/1456032851477… I was lucky to make and learn from that mistake pretty quickly with some algorithmic trading on much smaller amounts. With housing transactions being much larger and slower, you wouldn't learn this lesson until it was too late. Models never perform as well in practice as they do in theory, and you need to remember to account for both known unknowns and unknown unknowns.

A fascinating thread about Zillow, comparing it to quant trading mobile.twitter.com/0xdoug/status/145603… TL;DR: not taking into account adverse selection - other people's prices & trades aren't random/Gaussian, but adversarial (they're trying to profit at your loss) and better (they probably have better information than you)

That article doesn't really cover the important bit. Yes, the distribution changes, but that's not just something that happens, it's adversarial. mobile.twitter.com/0xdoug/status/145603… I'm quite sure there were people in Zillow's data science team who understood this, and maybe even tried to say, hey, this model we developed can't really be used for this new purpose you want to use it. But it's hard being that naysayer. I was at an interview with some people who were doing some things with machine learning on time series data... a topic I know little about, but enough to have heard of Eamonn Keogh and his many, many criticisms of how that's being done by ML practitioners with no time series background. What they were doing was pretty much everything on Prof. Keogh's "what not to do" list. But how do you tell them that? And how would it be if I'd already worked there for a while when you found out your company's business model has some pretty big flaws from a data science perspective? As long as they pay well, there's certainly a temptation to say, I only make the models I'm told to...

I think the title is highly misleading. The main point here is that Zillow simply had no idea what it takes to be a market maker and their pool was picked off by savvy traders. Good tweetstorms with technical explanations on how that happened: twitter.com/macrocephalopod/status/1455…... twitter.com/0xdoug/status/1456032851477…