I'm the dev that built Netflix's autoplay of the next episode. We built it first on the web player because it is easy to A/B test new features there. We called it "post-play" at the time.
When I worked there the product team at Netflix had two KPIs all new features were tested against: hours watched and retention. We would come up with all sorts of ideas to try out, and release them to small user populations of about 100,000 or so. It was great because you didn't have to debate much about whether a new feature was a good idea or not, you just built it and tested it. If the feature didn't increase hours watched or retention in a statistically significant way, the feature was removed.
Autoplay massively increased hours watched. I can't remember the exact numbers, but it was by far the biggest increase in the hours watched KPI of any feature we ever tested. There was some skepticism about whether the number was inflated by Netflix continuing to play when the user left the room.
As part of the autoplay test, we tested how long the countdown should be between episodes. 5 seconds, 10 seconds or 15 seconds. 10 seconds caused the biggest increase in hours watched. We thought that it gave people time to digest what they had just watched, but wasn't too fast (5 seconds) where it became jarring. Interestingly, Netflix recently changed the countdown between episodes to 5 seconds. That means they tested it out and found that people watch more if with a shorter countdown. This didn't use to be the case. Netflix user have become conditioned to expect autoplay.
So yes, Netflix wants you to spend more hours watching Netflix and the product team is scientifically engineering the product to make it more addictive.
But...the product team at Doritos does the same thing.
> When I worked there the product team at Netflix had two KPIs all new features were tested against: hours watched and retention.
So, I totally understand retention: that one makes sense to me, as that's revenue for Netflix; if I'm more likely to renew my account, then Netflix makes more money. But... I honestly don't understand why Netflix is optimizing for hours watched, and in fact I had assumed it would be the opposite (leading to Netflix actually making it kind of annoyingly difficult to watch the full credits on movies, for example).
> But...the product team at Doritos does the same thing.
This makes sense to me: the more Doritos I eat, the more money Doritos makes. This isn't true of Netflix: every time I watch something, it costs Netflix money. I would, naively, expect that what Netflix would want to optimize for is some weird balance of retention (at high priority) and least hours watched (at a much lower priority).
> So yes, Netflix wants you to spend more hours watching Netflix and the product team is scientifically engineering the product to make it more addictive.
I also appreciate that the more hours I watch of Netflix, the more I might be "addicted" to doing so... but the reason I'd expect Netflix to care about that is because it means higher retention, which is already being directly measured. Any other aspects of my addiction don't seem to benefit Netflix: it costs Netflix money every time I stream something from them...
...and, it would seem to me crucially, it means that Netflix is "on the hook" for more and more content. If, on a relatively short-term basis (such as during an experiment), my measured retention is the same, but my hours watched is higher, then it means I'm burning through content I like to watch at a faster rate, and so Netflix needs to spend more money to rotate their catalog and create more originals.
It just seems so weird that Netflix would be optimizing for "hours watched", and I'd love to learn from someone more on the inside who was involved in that why they would do that. I understand why YouTube and Facebook optimize for "engagement" (even though I also think it is abhorrent) as 1) their revenue is based on advertisements and 2) they spread using network effects; neither of these are true for Netflix.
> So, I totally understand retention: that one makes sense to me, as that's revenue for Netflix; if I'm more likely to renew my account, then Netflix makes more money. But... I honestly don't understand why Netflix is optimizing for hours watched, and in fact I had assumed it would be the opposite
Hours watched differences may predict retention differences farther in the future better than short-term measurements of retention, which may measure reaction to changes more than it measures the steady-state effect of the change.
If you want to act on A/B tests faster than directly measuring long-term retention would allow, having a proxy measure of long-term retention that mitigates the risk of optimizing for transitional rather than steady-state retention has value.
Unstoppable play-while-browsing plus the price hikes has me poised to drop it as soon as I finish two shows I'm near the end of. Probably in the next month. Either problem alone wouldn't get me over the "bother to cancel" speedbump, but I dread opening Netflix unless I'm browsing straight to something I already know is on there, these days. If it were still super cheap, whatever, but I can get a couple cheap services for its price now. Out it goes.
I'll just start picking it up a month a year or so to binge whatever I've missed.
Playing-on-the-menu and autoplay-after-episode would keep me from recommending it to older relatives. The last thing they need is their devices doing shit without their telling them to, especially with audio and video involved for extra confusing-the-hell out of them, doubly so when they're trying to direct it to do something (browsing the media selection). Probably direct them to Hulu, I guess, if they asked.
You can shut off auto-play junk on the menus? Didn't used to be able to. Great news, one of the worst UI features I've ever seen (as in, intended behavior, not a bug).
Why hours watched? Because Netflix wants as much of your time as you're willing to give. The more ingrained they are in your life, the less elastic your demand. We've seen them raise prices and remove just about all of the worthwhile content, and they have more subscribers than ever. Stay tuned for more of this.
I wonder how many of those extra hours watched are because the stream just didn't stop after the person fell asleep. I can't tell you the number of times I've fallen asleep while Netflix played (or walked away) and it went right into some other show I didn't care to watch (and never actually did).
If hours streamed is your metric for engagement.. of course having the stream continue on its own would make that number rise. It doesn't actually say anything at all about real engagement.
There was no “cost” they could have paid Disney (and now Fox), Warner, or NBC Universal to keep their content when they wanted it back to start their own streaming service with exclusive content.
I'm not on the inside, but I have a possible explanation.
Netflix has competitors. If we assume that leisure-time hours are fixed in the short term, the more hours you spend watching Netflix, the less you have available for Hulu, Amazon Prime, CBS, HBO, MLB, or Disney streaming services. Thus, if all those have comparable prices, more hours watching Netflix translates to greater subscriber-perceived entertainment value for money spent on watching Netflix than on the other services.
Netflix does not have to be number one, but it does have to be high enough on people's ranked lists to be above their individual fragmentation threshold.
If a customer's hours watched is low and dropping, Netflix is in danger of losing them as a customer, especially when Disney comes online later.
Perhaps if CBS All Access tracked that number, they would give up, and contract with Netflix or Hulu to stream their content. It seems to me that it is the service that would be least likely to get subscribers that are already at or near their fragmentation tolerance.
> Netflix has competitors. If we assume that leisure-time hours are fixed in the short term, the more hours you spend watching Netflix, the less you have available for Hulu, Amazon Prime, CBS, HBO, MLB, or Disney streaming services.
Not only those competitors but any other activity, from reading HN to working to going out with friends, etc.
We could lump those together as screen-time versus non-screen-time.
I hypothesize that converting an hour from one subcategory of screen-time to another is easier than converting it from non-screen-time to screen-time. I.e. it's easier to convince someone to watch Netflix instead of watching Hulu, than it is to convince them to watch Netflix or Hulu instead of hosting/attending a summer outdoor grilling party.
Netflix would show greater gains relative to competitors by converting more hours spent watching their streams. Any attempt to bring in more hours from the outside would also tend to help competitors, since they would then have an easier time poaching those hours from Netflix later. That might be beneficial in a nascent industry, but I think we're past that now. Nobody is going to be launching an explicit "stay in, and watch more TV" campaign any time soon, but apparently they all will whisper "now that you're already watching TV, why not just keep doing it?" via autoplay mechanisms.
And time spent working is definitely not leisure time, by definition.
This makes sense but I'm guessing there are diminishing returns that may actually turn negative at some point.
I.e. How much more likely is someone who watches 21 hours vs 20 hours to be retained? Then how much more likely is someone who watches 41 hours vs 40 hours to be retained? I'm guessing that on average there is a much smaller difference between 41 and 40 than between 21 and 20.
I'm also guessing it may turn negative as some people have shown they will subscribe for a month, binge on their favorite/new shows, and then cancel. Perhaps renewing in a few months time to do it again.
Would also be curious about other metrics such as search time : watch time. I'd imaging people spending more time searching and less time watching get frustrated and are more likely to cancel. With autoplay it eliminates searching for some.
I'm also curious why there isn't a setting to enable/disable autoplay.
There are probably so many angles to this but I will give one. Netflix is more likely to be used and loved if it gives someone something similar to the thing they just finished watching (like the next episode) then if they lose them to something they might not like by giving them the opportunity to find other content.
Also even if the content is not the next episode I feel I would be more likely to go with a suggested (new) video or content rather than something that takes me time to find (and therefore in theory happier with the service). For example there are so many choices on Netflix that I find I often might decide to not even take a look because of what I think the content is. [1]
[1] An example of this is the most excellent (now in S3) 'Money Heist'. The Dali masks were an instant turnoff for me that for whatever reason (rightly or wrongly) made me think it was some kind of fantasy rather than a crime drama (not saying my thoughts were right but that is what I thought).
While you watch Netflix, you don't watch a competitor. In a way, this autoplay feature removes a friction for the customer that doesn't have to decide what to watch next. This removes the "danger" that the customer think about that cool show that is not provided by Netflix.
At the end of the month, the customer may think about how much they use a TV provider and not the other, and decide to keep Netflix and ditch any other subscription they may have.
It's true that it burns through Netflix content but they made 1,500 hours of original content in 2018, content that they have to create anyway for other reasons, so a lot of your free time is already covered.
But isn't that just a proxy for retention then? Why optimize for hours watched which correlates with retention when you can just measure and optimize for retention directly?
Ah, I understand what you mean. Well, if you know variable X is a proxy for Y, but Y is a lot longer to measure than X, then I guess it makes sense to optimize X knowing that it will increase Y (if you are sure X will bring Y).
The cost produce something is sunk already. If everyone watches (birdbox) and talks about it can get more people to subscribe.
The more you watch the more likely you are satified. They won't make new content for you because you watch more. They know you will watch a variety of shows and will watch some less known content.
Plus view numbers are important for awards and privately they share with producers to encourage them to sell new content/partner up. No matter how much you pay artists still want the world to see what they created.
If I'm watching Netflix, I am fine with autoplay. Youtube too, while we're at it. I.e. if I'm there specifically to watch videos, and you want to autoplay the next video after I'm done, that's fine.
What I'm not fine with is places that autoplay that have no business doing it, especially if they do that specifically to brag about how many autoplays it has (i.e. CNN). CNN in particular is my main annoyance, as when I go there I just want to read their news article, NOT watch video, and yet I have to scramble to hit the pause button on their video the moment I launch one of their pages (even by accident, because I'm just clicking a google link or a Hacker News link).
So long story short; Netflix and Youtube or sites that are just about serving video (and nothing else), autoplay to the next video is fine and expected. EVERYTHING ELSE, DON'T AUTOPLAY.
> But...the product team at Doritos does the same thing.
This is a good comparison because we know it's a public health disaster. Unfortunately he junk food industry was powerful enough to repeal this kind of rules in their domain, and I'd actually be surprised if this one was enacted.
You wouldn't happen to know the guy/gal that implemented the automatic preview when hovering over a selection, would you? Curious as to how they felt about it, and how much that really affected retention (it bugs the everloving shit out of me, I try to use my old smart tv with an outdated version as it lacks this "feature").
I've never been able to understand why netflix benefits from autoplaying - you're not selling anything, you're just delivering content - so why does autoplaying, which presumably has some measurable, additional cost for bandwidth, end up benefiting the company?
Does the perceived value of netflix increase when autoplay is enabled? I really want to understand this, I've been strung up on this for a while.
When you have to choose between "watch the next episode" or "watch something else", Netflix benefits when the default is "watch the next episode".
When you have to consciously choose to watch the next episode, you can also make the choice to turn off Netflix and do something else. The more you watch Netflix, the more likely you are to keep paying for it. When you actively choose to do something else, the less likely you are to keep paying for Netflix.
It's not the perceived value, it's the choice.
When you have to make the choice, you could choose to spend more time with something else (like a video game, a Hulu show, etc...). If you choose to do that, you might get sucked into that game or show and decide to not come back to Netflix. Netflix would rather spend the few cents it costs to stream you a new video (which is probably already cached at your ISP) than make your choice to do something else explicit.
That said, I think it's a great feature that makes it easy to binge through a series. Its beneficial to both parties in that way -- I get to not move from my couch and they get to send me more entertainment (and keep me paying them).
Also -- the more you watch of a series, the more you'll talk about it to your friends. And the more likely they are to watch it. If you get stuck on Episode 2 of 10, you're not as likely to talk about it. But if you keep going to episode 3, 4, 5, then you are more likely to finish the series. And if you finish, you'll probably talk about it, and then the cycle continues...
The probability of you cancelling Netflix in a given month is inversely proportional to the amount of Netflix you or your household watched [1]. The expense of a marginal delivery of a particular episode's worth of resources is a great deal less for Netflix than the revenue of your subscription.
[1]: To a first approximation, anyhow. I bet there's a bump at zero, where people just entirely forget Netflix exists but they are still subscribed. If someone watches a single thing a month they're probably more likely to eventually realize they're not using it enough.
I suspect there's also an inverse relationship at the high end, because of people like me who'll subscribe for a month, watch all the new content, then cancel.
When you start measuring something, that thing tends to become what you optimize for, regardless of if it actually makes sense. Hours spent watching is some kind of indicator of whether people enjoy using the service, and is also a nice clean metric to be able to report on.
Serious question: how do you feel about having been involved in that feature, knowing that you were creating an addictive product and trying to get people more hooked on it?
You might try to say that people have free will and can do what they want, but you know that the reason it’s done is exactly because it’s addictive.
I left Netflix mostly because of this. I've always believed that too much TV was bad for you and I didn't like thinking about how to get people to spend more time on the couch. It doesn't feel great.
To me, Netflix is probably the most benign out of FAANG. I wonder how many people there are at the other companies wringing their hands over the effects of their work. Facebook in particular, what with the exacerbation of vitriol in the country, as well as potential for foreign actor misuse.
I respect you for leaving. I decided a long time ago that the boost in resume for working for one of these companies really wouldn't be worth the moral baggage (Netflix being a possible exception for the reason I stated above).
Did being responsible for a feature so important to the growth of Netflix result in a reward or promotion outside of your expected compensation?
And as a father with 3 young children, you don't know how much I appreciate what you built. Not having to click next between episodes of "word party" for my toddlers is sanity saving.
Also to the people without kids who will be judging this comment, you may not realize that many children's programs are not always 30 minutes long. Episodes of Peppa Pig can be ~ 8 minutes. And on iTunes you cannot make a playlist or autoplay episodes, making Peppa Pig during car rides on the iPad an experience akin to visiting Hell.
Now that my kids are old enough to control Netflix, it's awful all around. Needs much better parental controls. I suppose they don't do that because any useful ones would make their preferred metrics drop. Gonna feel good to cancel soon, for that and other reasons.
Are you condoning replacing children's autonomy and sidestepping parenting responsibilities with incessant dopamine depleting screentime/fostering electronic addiction at such an impressionable young age?
Why not read a book and play with some cool science toys and limit screentime?
I'm not a father but removing my children's autonomy with tech screentime and encouraging tech/media addiction worries me deeply...
Before having kids I thought EXACTLY as you do....
But its hard to explain exactly how difficult this is, and how little time discretionary time you have, especially if you have more than one child under five, and I have 3.
I'm not trying to avoid responsibility, but for example there are days when I really have to go to the bathroom at 5:00pm, and that just can't happen until 8:30 when I get them to bed. Also "Word Party" is as about as educational as a 2 year old can handle and assists in learning the alphabet.
And I think letting them watch educational programs during our commute the best use of that time I can think of. There isn't really a lot of cool science toys for when they are in a car seat.
Well, if it makes you feel a bit better, I love autoplay because most of the time I'm watching Netflix I'm cooking/cleaning/etc. I definitely clean less when there's nothing left that I want to watch, and autoplay means I don't have to break that flow.
Something I keep wondering about: For as far as I can recall there has always been something people are addicted to. In the 90s and early 2000s people were worried about TV time. Now that attention has moved around a bit: Netflix, facebook, instagram, YouTube, etc. So, who cares? Do people worrying about this have some grand idea that if humans legislated this addiction away then.. what?
Yeah, ever so often when I want to continue watching the current series I realize that autowatch made me watch the whole season while I was asleep on the couch.
Really wish autoplay would stop every three or four episodes.
I've been annoyed with it on series that have unique end credit songs per episode. They put that song in there and I'd like to hear it, but autoplay cuts it off and just starts the next episode.
> But...the product team at Doritos does the same thing.
I get awfully suspicious any time someone so clearly on today's side of the digital divide feels the need to hop the fence back to yesterday to elucidate an ethical argument. (And esp. on HN where you'd be hopping away from the place we're all standing.)
What's the threshold of addiction beyond which a digital streaming service becomes unethical in your opinion? How about dangerous?
I ask because you unequivocally refer to the product you worked on as "addictive." I can't imagine working on such a product without a clear sense of what the boundaries are.
When I worked there the product team at Netflix had two KPIs all new features were tested against: hours watched and retention. We would come up with all sorts of ideas to try out, and release them to small user populations of about 100,000 or so. It was great because you didn't have to debate much about whether a new feature was a good idea or not, you just built it and tested it. If the feature didn't increase hours watched or retention in a statistically significant way, the feature was removed.
Autoplay massively increased hours watched. I can't remember the exact numbers, but it was by far the biggest increase in the hours watched KPI of any feature we ever tested. There was some skepticism about whether the number was inflated by Netflix continuing to play when the user left the room.
As part of the autoplay test, we tested how long the countdown should be between episodes. 5 seconds, 10 seconds or 15 seconds. 10 seconds caused the biggest increase in hours watched. We thought that it gave people time to digest what they had just watched, but wasn't too fast (5 seconds) where it became jarring. Interestingly, Netflix recently changed the countdown between episodes to 5 seconds. That means they tested it out and found that people watch more if with a shorter countdown. This didn't use to be the case. Netflix user have become conditioned to expect autoplay.
So yes, Netflix wants you to spend more hours watching Netflix and the product team is scientifically engineering the product to make it more addictive.
But...the product team at Doritos does the same thing.