So that we don't have to relies on a platform which is infamous for automatic ban anything it's automated system think inappropriate. Like now, if you utter the word coronavirus, you may get banned regardless of the content.
It's more likely to allow accounts that have lots of flagging to be automatically shunted to an automatable verification step. Accounts that pass verification can then be worth human attention. Accounts that do not attempt verification end up muted.
Seems like a reasonable way to scale human intervention in the face of cheap, broad usage of throwaways by trolls.
You can do A/B tests to figure out what sorts of behaviors ride the line of triggering bans vs what kinds of behaviors result in an outright ban. It lets you optimize for just-trash-enough to not get banned, which is not a desirable end state for users.
1. It wouldn’t be 5 minutes. This could easily take hours to resolve.
2. Even with human support, companies like this would continue to be incentivised not to share what exactly triggered the ban in order to protect their systems from spam and abuse. You can’t provide services to billions of people without automation.
3. Humans reviews would be performed by humans. You’re replacing an imperfect system with another imperfect system except now things are slower and cost more.
As a deterrent for abuse, it makes sense to suspend lots of accounts up front, pending investigation, and then let them back selectively as they are reviewed slowly. But if you're not doing the review, it makes no sense to ban lots of users while not addressing the root cause. That's just a way to run out of users.
- Transparent and consistent moderation policies. Bans are inconsistently applied for reasons that are often hard to figure out.
This is one of those things that sounds great in theory. But in reality, it likely functions much more like any other adjudication system. People can break rules an infinite number of ways, so there are an infinite number of reasons to ban an account. Understanding many of them will require context that may be problematic to share (for ex: if it happens in DMs). So in practice, the shorthand public reason for bans will pretty much always reduce to "violated site policies."
Fully agreed - I wasn't suggesting they review every manual decision, but do it as part of a proper appeals process. Google, for instance, is likely seeing abuse on a scale where they have to have automated bans - for everyone's benefit! - and there's nothing fundamentally wrong with that as long as you can escalate to a real human.
Edited the comment to reflect this - thanks!
I worked on an abuse prevention system in the past and know the challenges very well, except my company actually put in the effort to respond to every appeal and compensate affected customers for their troubles.
Yes, humans are expensive, and spammers will try to game the appeals process, too - but it's simply a cost of doing business.
We'll quietly ban serial trolls or outright spammers, but otherwise we provide an explanation so that the community can benefit from observing the application of the guidelines. We have other tools for dealing with accounts created as a result of the apparent ban.
I like this idea too. Also temporary bans are good for offering a cool-down period for users who generally are constructive but tend to lose their tempers from time to time.
reply