- Joined
- Aug 28, 2019
Discord, big data, and pedophiles - The application of data science to Discord degeneracy
- Thread starter grand larsony
- Start date
- Joined
- Dec 28, 2014
- Joined
- Oct 20, 2018
This is exceptional work, OP.
@grand larsony I wouldn't feel too bad if the bot brings up false flags - it's normal. The fact you can pull 10 messages and around +40% of them being related to sexualising children is enough to show that it's capable of exposing the people who are a threat. Language is very nuance and even the most money pumped AI still struggles with it.
@grand larsony I wouldn't feel too bad if the bot brings up false flags - it's normal. The fact you can pull 10 messages and around +40% of them being related to sexualising children is enough to show that it's capable of exposing the people who are a threat. Language is very nuance and even the most money pumped AI still struggles with it.
RVNX
kiwifarms.net
- Joined
- Oct 10, 2017
This is nuts. I'm also not surprised that Discord doesn't really care much about these sort of things, given that a lot of upper-tier Discord staff are also sexual deviants of their own. And like you said, it would involve a massive legal circus that nobody really wants to handle. I'm also pretty sure they don't do it on their own so they can blame PhotoDNA on anything that slips under the radar.
- Joined
- Sep 5, 2023
Generally, you should operate under the assumption that they do, but Discord is actually decent about how they hand out permissions to bots. If your bot is in <100 servers you can do basically whatever you want with it and Discord's interference in your activities is much smaller. Once you pass 100 servers and have to do ID verification stuff, they also make you apply for different permissions. For example, I had to email back and forth with a Discord staff member explaining why I wanted permission for my bot to see messages that didn't explicitly tag it by username. I had to explain what I'd do with that permission, how I'd store the data, who would have access to it, etc. When I told them I was running it off of my home computer (no longer the case, btw) they actually asked me to explain what it'd take for someone to walk into my house and steal my hard drive. Like, I had to tell them how hard it'd be to rob my house lol.
From what I understand, they really cracked down on bots indiscriminately harvesting user info after they had some issues where bots were built for the explicit purpose of making semi-public community data available to the public. Like there were a couple of bots where, you'd add it to your server, and without telling you, the bot would make all of your server's messages + user info publicly searchable. It seems that caused them a bit of embarrassment.
- Joined
- May 9, 2017
There was an understanding that they would limit themselves to adult booty.
That understanding has been violated.
- Joined
- Mar 28, 2023
This is incredibly impressive
- Joined
- Dec 12, 2022
There's nothing wrong with making it more convenient by bringing it all into one app, for free, either.
You're being purposely obtuse with these kinds of claims. The full feature list, convenience and accessibility of discord are what brought it into the big leagues of gaming-oriented chat programs, which naturally brings in children and zoomer type features. Making the claim that the application itself is developed with the intent of providing avenues for preying on children is a leap that needs more evidence than literally adding features that normal people want in a messaging application.
- Joined
- Sep 29, 2016
@grand larsony How much overlap is there between the pedo users and users talking about other topics likely to be the subject of moderation, like animal abuse or drug use?
- Joined
- Sep 5, 2023
I think that a lot of times, people want to believe that someone is in control. Even if someone malicious is in control, the idea of someone malicious pulling the strings from behind the scenes is more comforting than the idea of total chaos stemming from the disinterest of the people powerful enough to make a more orderly, sane, positive world. At least if that were the case, there'd be someone who could be held accountable for the things that are wrong with the world. I can see why it's tempting to believe that this is intentionally abusive behavior from Discord higher ups, even though I know the reality is that they just don't care.
This is harder to answer since the moderation API covers more broad categories. As the bot grows I plan to make purpose-built classifiers for more fine categories. For example, I'd like to separate the "hate" category into racism, sexism, transphobia, separate the sexual category into normal sexuality, fetish content, extreme fetish content, etc. But to do that I need more training data. 2 million might sound like a lot of messages, but the vast majority are people saying totally normal things even with as many disgusting freaks as there are in the database, so the positive examples I have of specific forms of inappropriate content are somewhat hard to come by at this stage.
If you're curious, here's the list of categories I currently scan for: hate, hate_threatening, violence, violence_graphic, sexual, sexual_minors, harassment, harassment_threatening, self-harm intention, self-harm instruction, insult, obscenity, threat, identity_attack. There's some overlap between categories but the points where they don't overlap allow moderators to make some nice fine-tuned choices about what they'd like to allow. E.g. setting a strong obscenity filter but a weak insult filter, so that "you're so dumb" wouldn't be blocked but "you're a fucking retard" would.
Good news though - Meta just released a new classification tool that's purpose built for matching against user-defined categories. This is something I plan to add to the bot as soon as I get approved for research access. This will make it much easier to scan and track a much wider variety of content categories. Facebook spyware link, click at your own risk lol - https://ai.meta.com/llama/purple-llama/
(I mentioned the hate category here, but before anyone jumps down my throat about it, I think that yes, free speech is critical to a free society, but there's also plenty of times and places where certain things are obviously unwelcome. Kiwi Farms is a great place to call people niggers and trannies, but the local PTA meeting probably isn't, that's my point.)
- Joined
- Jan 14, 2023
Holy shit, that's impressive work, man.
Aren't you scared to post sample messages? Call me paranoid, but if anyone recognizes them, or worse, can search for them in their own databases, your bot could be traced.
Aren't you scared to post sample messages? Call me paranoid, but if anyone recognizes them, or worse, can search for them in their own databases, your bot could be traced.
- Joined
- Aug 14, 2022
I think this touches on my point but does not really answer it in some ways. I understand your strategy when it comes to entering the market and building trust, but I am more curious about the economics of it all on both sides.
I think that you have the right idea, but that there might be more to it than moderation. The way I think of it, it might actually be your foot in the door to greater things.
If I own a discord and I pay you to help with moderation or admin, then I am making money one way or the other. And moderation is only a mean to an end.
Now, if you know how I make money, your bot can become insanely more valuable.
If you are somehow able to profile people, posts or whatever the fuck they are doing on discord and cross this data with conversion metrics, you can deliver very valuable insights. I am talking an easy sell for like $499 a month.
- Joined
- Sep 14, 2021
They could, but that would require the senders of these messages to out themselves as the ones doing pedophilic mommy breastfeeding roleplays on Discord. It’s possible that nobody who has access to these databases and could do the reverse lookup you’re suggesting will do so, so that the spotlight is never turned on them.
The Tall Man
kiwifarms.net
- Joined
- Aug 4, 2022
It's an estimated 20billion dollar industry.
Child exploitation is a Black market, that means a lot of it is run by the Mob that means the Glowinthedarkniggers are in on it. The Mob and Glowies blur together. Black budgets don't get directly funded by Taxpayers, they are made off the books.
They are not just disinterested in fighting child exploitation, they are opposed to it, especially at the top.
- Joined
- Mar 1, 2023
- Joined
- Feb 3, 2023
I always had a feeling the guy who coded Dyno was based
More like Five-year-old Body Inspectors
- Joined
- Aug 2, 2021
@grand larsony This whole endeavor could do a lot of good for a lot of people. Thanks for doing what you're doing and for sharing it with us here, we need more users like you.
- Joined
- Jun 30, 2023
Then what is the SEC?
I have had a lot of success with using classifiers to train themselves. I start with particularly strong examples (find them using keywords and the like), train the classifier off those, and use that trained iteration to identify more subtle examples, which I can fix or filter manually. It doesn't take long to train text-based classifiers, and you only need a few decent iterations before the accuracy gets pretty good. Then you could always adjust sensitivity with a bit of quick maffs.
Also, great work. I started maintaining a few Telegram bots years ago, not necessarily for moderation. Once they began getting some use, I quickly realized—much like yourself—that I can use them to get insights into fringe communities. For better or worse, we all know Telegram is very hands off, so we don't have a lot of people monitoring this sort of thing. I didn't want to collect any images for obvious reasons, but it only took ~30 minutes of logging message text until I had been scarred for life. Curiosity killed the yat, I guess.
I was going to shut it down™ until I realized how many researchers would kill for this data. Now I just sell it in bulk and use some of that money on the server costs. Very lucrative, and I love the intense irony of Telegram users who think their messages in public groups are even remotely private having their chat data sold. If they ain't broke nowadays, I don't touch any of the bots with a 100 foot pole.
derpherp2
kiwifarms.net
- Joined
- Jan 3, 2020
I support the idea that you need to get this published academically by the way behavior patterns on the internet WAS a hot topic when I was there. Idk about now, but we will need this shit in the future if/when people start going after groomers again.