Working on a decentralized forum - Python, Tor, no blockchain

[hundredpercent]

In the best of all possible worlds, they would just leave us in peace. But they won't.

This site is great, and definitely more robust than it was two months ago[1]. However, I still think there is a value in decentralized forum software. There are two principal reasons:

1. Events may change again, and the Internet is hardly guaranteed to be free forever.
2. It would be good to have access to a 100% robust "bunker" that you can always rely on - not just for this site, but also for other communities.

For these reasons, I've decided to stop shitposting and to make a real life effort post. Thus far, it's not anywhere near ready for general use - I just wanted to make a thread and post about it for those who are interested and to keep myself accountable.

My basic design goal is to reimplement FMS[2][3][4], but using modern technology. That system is decentralized, spam-resistant, and well-tested. (Technical explanation follows, but those links are recommended reading).

I don't have an implementation that is usable yet, but here are my notes on the design. I appreciate all critique.

Design

Nodes

Each user runs a node. A node consists of:

• An ed25519 keypair.
• A manifest[5], which contains the following data:
  • A list of posts created by that user
  • A list of threads that user has posted in.
    • The user need not have made those threads.
    • Those threads are digitally signed, again, by their author.
  • A list of transport endpoints; URLs at which new versions of this manifest may be found.
    • These will have authoritative=true set for own onions and authoritative=false otherwise. This is mainly to enable other nodes to know where to expect to find their manifests (and add as non-authoritative sources) if they are trusted by that node.
  • A list of trusts. This basically merges together the Ratings and Reports features of XenForo/phpBB/etc; see section "Web of trust" for more details
    • It might be valuable to "freeze" users at a specific version. I have not thought about the details of this and it will not be an initial priority.
• A signature to the manifest[6], which contains the following data:
  • Ed25519 signature
  • A sequence ID (strictly increasing); a more recent manifesto shall have precedence over an older.
• An onion (optional), which hosts own manifests as well as those to whom that user subscribes.

Posting

On making a post, the following actions will happen:

1. A local entry will be written to the posts database, but missing some content of a technical character.
2. The missing content will be filled in (see routine normalize_outbox()).
3. Assuming the contents of the manifest has changed from the last published version or there is no last published version, a new manifest will be generated, serialized, and saved into local database. Such manifests will contain all posts made by user, in addition to all the threads[7] they are posted within.
4. Node pushes manifests to all nodes which it trusts and which trusts such node over onion.
5. Node exposes new manifest on its own onion.

On making a thread, the same procedure will happen (a thread by necessity has at least one post, the OP), but a thread will also be added to the manifest.

Web of trust

All fora require moderation. This is not a problem; the problem arises when that moderation is not based on consent. If you want to see viagra spam, death threats, etc, that's your problem.

Web of trust-based systems handle this in a similar fashion to the Internet. I'll try to explain it with numbers schematically:

1. I trust Alice 80%
2. Alice trusts Bob 70%
3. Bob trusts Carol and David 30%
4. You trust me 50%.

You therefore transitively trust Alice, Bob, Carol, and David to some degree. If you disagree with this, then either stop trusting them or stop trusting the person which assigned trust to them. To be specific, your locally seen peer trust will be as follows:

1. You trust Alice 40%
2. You trust Bob 28%
3. You trust Carol and David 8.4%

It's worth noting that this can, with some effort, be seen basically to be the way the Farms operate today:

• Null trusts staff 80% or whatever
• Staff trusts new users 5% or so on joining
• If staff "stops trusting" a user, that user is banned
• If Null "stops trusting" a staff member, he is no longer a mod.

Technical note: There are two types of trust. Message trust, which goes from -1 to 1, and trust list trust, which goes from 0 to 1. Message trust is a judgement about how much you want to see posts by that user, trust list trust is a judgement about how much you want to trust that user's judgements about other users.

Technical note: Message trust isn't calculated by naively multiplying down the path, since this is trivially Sybil-resistant. I intend to use personalized PageRank with self-trust, but I appreciate suggestions regarding this. (TK! There's two papers about this that I can't find)

An optimization to this system is to treat (non-sage) replies to posts as a (weakly) positive signal. This makes the trust graph denser, and appears to work fine in current systems. (Compare KF, where ~everyone is within 2-3 reply/rating degrees of staff members)

There is also a Sybil resistance mechanism described in a paper (TK!) to penalize the TLT of nodes that incorrectly rate other nodes; this is worth looking into but not essential.

For more details regarding the operation of Web of Trust systems, see the literature on FMS linked in the footnotes.

Probably, the earliest MVP will just have a basic subscribe list.

Reading

You download manifests from all the nodes with peer trust above a certain epsilon, and that pass validation. You then insert all manifests into the database with non-negative message trust (or message trust above a certain epsilon, up to you. The intent is to show posts from new users but to hide new threads made by them and to treat their posts as sage)

Technical note: All rows are fk'ed to the node that is responsible; inserting a new manifest is a simple matter of deleting all rows of that node, inserting, and then letting the COMMIT take care of FK based trigger deletions.

This means that you only download content that you want to see. It's not possible to flood, since you are only ever pulling content, not pushing.[8]

Registration

Registration is done by finding any trusted node and getting them to assign you a non-zero trust.

As an implementation note: It would be easy to create bots that accepted a CAPTCHA and assigned a trivially low but not zero message trust, so that new users could join.

This basically mirrors, respectively, the invite system of KF and the registration system of KF. The difference is that anyone could run either service, and that users will not be dependent on their initial introduction point once they have made a few (non-spam) posts.

Other notes

I deliberately did not want to use too much complicated technology; decentralization is already hard enough as it is to get right, and using absurd Rube Goldberg machines hardly makes things less error-prone. There is no technical reason to use blockchains for this, which would only make everything by far worse.

Manifests are kept in JSON, since that's pretty much a universal standard
The code is in Python, since everyone knows it and it has a lot of libraries.
Things should be kept simple so that everyone should be able to contribute. This is important:

• It's important to keep a big potential contributor base
• There are unlikely to be any salaried full-time devs
• There aren't that many technologically proficient KF users worldwide

Roadmap:

• Mild refactoring:
  • Change table names to be unique
  • Merge all sqlite databases
• Sharing of peer manifests:
  • Add new table peer_manifests(id, seq, blob)
  • Add new table my_manifests(seq, hash)
  • Decide on serialization format (data + sig [+ seq + id])
  • Encode and sign manifests
• Expose known manifests on HTTP over To
  • Run a(nother) local HTTP server
  • Preferably over Unix socket
  • Integrate with the Stem controller library
• Subscribe to others' manifests over Tor
  • Strictly opt-in basis at first, like RSS
  • Onions only. I don't want for there to be a possibility of a proxy leak.
• UI features for trust ratings
  • Neg/posrate
  • Automatic imputation by non-sage replies
  • Notes feature
• Implement calculation of trust rankings
  • Probably personalized PageRank with self-trust (probably using networkx library)
• Automatically subscribe to users with positive trust
• Other work
  • Sybil resistance
  • DoS resistance
  • UI work
  • maybe public node support some day, if I get that far

This is some amount of work, but a lot of it is trivial. All contributions are encouraged! In particular, here are some areas that I am not very good at:

• Web development/web design. I just have a very rudimentary table-based imitation of "classic forums" (i.e. phpBB) - see screenshots. If you can submit CSS or templating changes that makes it look nicer, this is welcomed. Also CSRF/XSS protection and that general sort of thing.
• Interface design. In particular, constructive suggestions on how to make WoT user-friendly, by giving it an UI similar to negrating on XenForo.
• Packaging, in terms of distro compliance, creating systemd/init system files, and porting to Windows. In the short term, just running a .py file is Good Enough(TM) for the few autists (o7) who want to try it.
• Tor, in particular Stem/control port protocol and (later on) DDoS protection - think ClientAuthorization but only for mutuals.
• Abstract elements of computer science, in particular graph trust algorithms for TLT. Is there a theoretical justification for personalized PageRank *with nodes assigning non-zero trust to themselves*?
• Scaling.
  • How many O(n^2) algorithms are there, which ones are needed?
  • Three months' retention of 100 PPH (KF does ~700) averaging 1000 chars at 1:8 compression ratio is 25 MB of storage - how close can we get to this lower bound, and what optimizations would it take?
  • Probably user count is a bigger issue, but then again most users (at least on KF) are idle.
  • Scaling is a good problem to have, since it'd mean people are using it a lot, which I consider unlikely.
• Cryptography - how to sign manifests in an interoperable fashion? Is the signature mechanism used for threads workable, is there a cleaner way? Are there (serious) downsides to signing raw blobs without using something like HKDF first?
• Data interchange formats. How do you design something that's future-proof but not underspec'd?
• Database schema design (SQL)
• General programming/HTML - there are some endpoints that need to be fleshed out and templates written for them

Repo is at https://gitgud.io/hundredpercent/zboard. Software is not yet ready for general use; don't download this and expect a smooth shitposting experience.

[1]: This is totally different from SneedForo AKA RuForo, which aims to be a XenForo replacement for the continued use of this website. My goal with this project is not to be schismatic.
[2]: https://blog.locut.us/2008/05/11/fms-spam-proof-anonymous-message-boards-on-freenet/
[3]: https://fms.fn.mk16.de/operation.htm
[4]: http://freesocial.draketo.de/fms_en.html
[5]: As a performance matter, it may turn out to be a better idea to break these up, but I haven't done the numbers yet.
[6]: While the main body of the manifest will be JSON, I haven't decided yet whether the signature should be appended onto or downloaded separately from it, or by what format that should be. The signature itself will be made according to BEP 44. I appreciate all pointers here. I'm also not sure whether sequence IDs should be sequential or based on the time.
[7]: Not a full copy of them, just time + OP + subject. Otherwise, if user A posts in a thread made by user B, and user C is receiving posts from A but not B, user C would not know the subject line of that thread.
[8]: Aside for the optimization mentioned earlier, but for any of that to happen you have to have assigned trust to them already.

 

[Hathungor]

This does indeed sound very interesting. Unfortunately I'm just a dabbling laymen so I highly doubt I'd be able to contribute anything of value, let alone understand all the intricacies.
Something like this is badly needed, but any such project so fat has gotten mired by crypto BS.

A few things to consider. Excuse me if this was already addressed above:

Since the user has no control over what other trusted users post and may not even be aware of all of it, and since some of it may be illegal to host where they live, is it possible to introduce a degree of randomness as to what's actually hosted on which nodes?
Maybe only 70% of nodes who trust you would actually end up hosting a copy of a post it themselves, just a meta-manifest of the whole thread and a pointer to anbother node that does have a copy of the post.
Further, the hosted information should be encrypted and only available through other nodes and never locally accessible. That way there's a degree of plausible deniability if cops ever kick in someones' door.
All posts and media are encrypted, cannot be locally decrypted and there is plausible deniability as to whether the offending illegal speech is actually hosted on the machine that was used to access the service.

Also, software for a non-participatory "blind" node might be a good idea, similar to how TOR functions already. Universities and such can run it to support up to x communities with y amount of resources, bandwidth and hosting space on a highly trusted node, but anyone hosting such a node would have zero control over, insight in what communities actually use it or ability to actively participate through that particular node. This might be interesting if this thing is ever finished and ends up being used by other communities than Kiwifarms.

 

[hundredpercent]

Thanks for the feedback, I'm glad to hear it, will respond inline.

This does indeed sound very interesting. Unfortunately I'm just a dabbling laymen so I highly doubt I'd be able to contribute anything of value, let alone understand all the intricacies.
Something like this is badly needed, but any such project so fat has gotten mired by crypto BS.

Yes. Cryptocurrency is useful, as currency, but the situation where you have to shoehorn it into everything since you make more money by promoting a shitcoin is simply untenable,.

A few things to consider. Excuse me if this was already addressed above:

Since the user has no control over what other trusted users post and may not even be aware of all of it, and since some of it may be illegal to host where they live, is it possible to introduce a degree of randomness as to what's actually hosted on which nodes?

My intention is that users only host what users whom they directly trust post. I don't think it's possible to violate the law that badly since it's only text, but my idea was to add some feature like "do not mirror content for X users" later on.

Legally speaking, you are fine - section 230 and EU E-commerce directive explicitly prohibits liability for hosting of content you are not aware of, so it would only be an issue if you are served notice of it, at which point it would be trivial to delete. Besides, the whole thing is on Tor.

One thing you could do is to just not host anything but your own posts - it wouldn't hurt things very much. Another would be to have one onion service per node you are mirroring. Implementing this would be annoying, but this would make it ~impossible to see who hosts what.

That being said, I'm not sure this is a problem. Sensitive users (along with phoneposters) can just refuse to run an onion outright - it isn't really required for anything to work, as long as some tiny fraction do it.

Maybe only 70% of nodes who trust you would actually end up hosting a copy of a post it themselves, just a meta-manifest of the whole thread and a pointer to anbother node that does have a copy of the post.

That wouldn't be needed - if the node says it does not have it, it shouldn't be listed as a transport endpoint in the first place. So there is no need to distinguish between "I don't have it but it exists" and "I don't know anything".

I guess you could have separate storage of transport endpoints and posts, but it seems like a lot of work for questionable gain.

Further, the hosted information should be encrypted and only available through other nodes and never locally accessible. That way there's a degree of plausible deniability if cops ever kick in someones' door.

This would be quite difficult. Imagine you write a post on here - are you supposed to be unable to see what you just wrote?

Also, it would be trivial for the police to run a node and just log all communications - if it's a public forum, that's not possible to avoid.

All posts and media are encrypted, cannot be locally decrypted and there is plausible deniability as to whether the offending illegal speech is actually hosted on the machine that was used to access the service.

This would be technically difficult, since it has to be accessible by everyone. As far as I know, there's very little content that's illegal to host anyway. CSAM and copyright, yes, but that's all media. Hosting hate speech with no insight into the content isn't illegal in any country that I know of.

Also, software for a non-participatory "blind" node might be a good idea, similar to how TOR functions already. Universities and such can run it to support up to x communities with y amount of resources, bandwidth and hosting space on a highly trusted node, but anyone hosting such a node would have zero control over, insight in what communities actually use it or ability to actively participate through that particular node. This might be interesting if this thing is ever finished and ends up being used by other communities than Kiwifarms.

You would, by virtue of how the system works, be able to see the content hosted on it. This is unavoidable, since you have to be able to share it with others. "I'm going to share the password to this document to everyone except for myself" isn't possible, unfortunately.

It wouldn't be difficult to do technically, but it seems even easier to just have a dummy node that doesn't bother to post but has high trust list trust.

Anyway, hosting isn't a big problem. I have made 466 posts - if they average 1 kb each (and they don't), that would be about half a megabyte. So you could fit a few years of reasonably active posting by a few thousand users on a single 4GB flash drive.

Like, without images or anything, text is very small. If you have a 10 Mbit/s Internet connection, you can download enough reading material to last you the rest of your natural life in a few minutes.

I think the big scalability issue is going to be the fact that you have to locally store everything you want to be able to read (this is the problem that Bitcoin has too), not the bandwidth. I don't think this is insurmountable with good engineering, but it's going to be annoying and unpleasant to deal with.

My personal take is that scaling shouldn't be a massive concern, since if you start getting thousands of users posting very rapidly then the project has already on some level succeeded, and if so you can deal with those problems then.

Thanks for the feedback, I really appreciate it!

 

[Hathungor]

I wasn't aware that this was going to be text only, I assumed (heavily size restricted) media was part of the plan.

You're not just contending with EU and US law though. If this does indeed take off and get used, you don't know who and under what circumstances will be using it to what purpose. That's why I thought not being able to locally decrypt the hosted content would be of value. You don't have to fetch your own messages locally after posting them. Once they're distributed you can retrieve them from other nodes.

Say 10 years from now there's an uprising in Iran or something. Your decentralized forum software gets used by anti government forces. If they legitimately don't know what they're hosting and it can't be locally decrypted it would make things a LOT safer against government belligerence, even in a rubber hose scenario.

 

[hundredpercent]

I wasn't aware that this was going to be text only, I assumed (heavily size restricted) media was part of the plan.

You could add it, but it would have to be a mostly separate thing - shipping multi-megabyte manifests would get out of hand fast. I think I would rather wait for Tor to support UDP and then add torrent support, frankly. Either that or DDL from a self-hosted onion á la OnionShare

You're not just contending with EU and US law though. If this does indeed take off and get used, you don't know who and under what circumstances will be using it to what purpose. That's why I thought not being able to locally decrypt the hosted content would be of value. You don't have to fetch your own messages locally after posting them. Once they're distributed you can retrieve them from other nodes.

Yes, but you still have to store them on your hard drive at some point to read them. So I'm not sure what the dance of post-delete-download actually gets you. And to be able to read the content, the key has to be public, so the encryption is pointless in the end.

Say 10 years from now there's an uprising in Iran or something. Your decentralized forum software gets used by anti government forces. If they legitimately don't know what they're hosting and it can't be locally decrypted it would make things a LOT safer against government belligerence, even in a rubber hose scenario.

What's the point? Either there is anti-government content on there, which you would be able to see by just ... opening their computer and reading the posts (since if you couldn't do that, it wouldn't be a very good forum), or there is not, in which case there is no legal problem anyway.

The threat model here is that someone gets v& for entirely unrelated reasons, and then they look at their hard drive and determine they're wrongthinking, right? If so, what more do they need than their private key (which, by virtue of cryptography, is required in order to post) and the publicly available posts?

I think it would make much more sense in such a scenario for them to either just use solid, reliable full-disk encryption (VeraCrypt, LUKS) with rubberhose encryption, or to use different services altogether (Ricochet, Signal), than to try and concoct a plausible deniability scheme on the application level for a public forum. (Such schemes haven't turned out great in practice - see the many disastrous attacks against Freenet in Opennet mode and compare it to Tor's track record)

 

[$80 Nude Haircut]

I can tell already based on how complicated it sounds this would never catch on with any substantial amount of people.

 

[Kosher Dill]

I think FMS-style "forums" (which are not actually forums at all) are just no good. Replacing moderators with the "web of trust" sounds like a simple swap, but it's actually throwing out any coherent notion of community. What's the point of saying "I, a highly trusted moderator, ban User X", if User Y can just turn around and say "No, actually I no longer trust you as moderator and I'm going to keep sperging at User X forever and anyone who can hear me will hear it"?
Or similarly - "This post is deleted" / "No, you're deleted to me".
One might argue that most people will just go with the flow, but keep in mind that anyone using an experimental distributed forum is likely to be the kind of obstreperous nerd who will customize their trust list just how they want it.

To keep discussions from going off the rails I think it's important to have one authoritative view of what is "in" and what is "out".

 

[⠠⠠⠅⠑⠋⠋⠁⠇⠎ ⠠⠠⠊⠎ ⠠⠠⠁ ⠠⠠⠋⠁⠛]

I can tell already based on how complicated it sounds this would never catch on with any substantial amount of people.

How complicated would a description of XenForo sound?

This basically sounds kinda like 'Secure Scuttlebutt', which is one of the only federated social media type thingies that actually works (rather than being a source of ever-increasing requests from the folks running sites like poa.st for donations to chuck hardware at shitty software).

It is true that there is a barrier to entry for this. Hopefully, that could be handled with a nice simple installer to spin up your local node.

I think FMS-style "forums" (which are not actually forums at all) are just no good. Replacing moderators with the "web of trust" sounds like a simple swap, but it's actually throwing out any coherent notion of community. What's the point of saying "I, a highly trusted moderator, ban User X", if User Y can just turn around and say "No, actually I no longer trust you as moderator and I'm going to keep sperging at User X forever and anyone who can hear me will hear it"?
Or similarly - "This post is deleted" / "No, you're deleted to me".
One might argue that most people will just go with the flow, but keep in mind that anyone using an experimental distributed forum is likely to be the kind of obstreperous nerd who will customize their trust list just how they want it.

To keep discussions from going off the rails I think it's important to have one authoritative view of what is "in" and what is "out".

I do think the 'web of trust' thing is unnecessary and a means for trusted moderators to explicitly delete posts on behalf of a forum 'owner' is more important, but it sounds like you'd have concerns about users using the 'ignore' feature on regular forums too.

As much as crypto sucks balls, if you aren't doing email verification, you probably need some sort of proof-of-work method to reduce the rate at which people can spin up accounts to troll.

 

[Stabmaster Arson II]

I've thought about this a lot, and I don't think it would work.

Text only would be able to make someone have a complete copy of course, so that rules out the hard drive space issue. But any splitting up of hosting to "I trust this user, I don't trust this user "would just end up fracturing and losing posts.

Sometimes awful posters make good posts, even a stopped clock and all that. It would fracture things too much, and there would eventually be data loss.

 

[Kosher Dill]

but it sounds like you'd have concerns about users using the 'ignore' feature on regular forums too.

Ignoring doesn't fragment the consensus view of what is or isn't "on the forum", that's the difference. And on a regular forum, ignoring is limited to one specific form: not displaying a specific poster's content to you. You can't ignore an edit, deletion, ban, etc. And what you ignore doesn't have any bearing on others.

But any splitting up of hosting to "I trust this user, I don't trust this user "would just end up fracturing and losing posts.

Yes, or even just unpopular/obscure content dropping out of hosting because it's not "on anyone's list". Maybe a guy with one post in an old thread will just vanish from the record. I think this is inevitable with any sort of IPFS/Freenet-style "host what you want" strategy. It can be worked around, but doing so would defeat the purpose of having distributed hosting at all.

 

[hundredpercent]

I can tell already based on how complicated it sounds this would never catch on with any substantial amount of people.

It's quite complicated on the implementation side, but not to use. There, I would rate it about the same as uTorrent or what have you - mildly annoying, but doesn't take a genius either.

The UX for end users would be:

1. Download Windows installer
2. Install
3. Navigate to http://localhost:1488
4. Solve a few CAPTCHAs or get an invite
5. Start posting

You could theoretically make public web proxies (would be way less liability and management than running a forum), but obviously that's far ahead. That said, if you did it, the experience would be

1. Navigate to https://your-favourite-instance.biz
2. Read posts instantly
3. To post, register more or less as on a normal forum

It would be a bit of work for the host to set it up, but I think the theoretical upper bound for "end user experience" in that case would be very, very close to "centralized forum" tier (basically bounded by quality of HTML frontend)

I think FMS-style "forums" (which are not actually forums at all) are just no good. Replacing moderators with the "web of trust" sounds like a simple swap, but it's actually throwing out any coherent notion of community.

Thank you very much for the thoughtful critique - I will respond inline.

What's the point of saying "I, a highly trusted moderator, ban User X", if User Y can just turn around and say "No, actually I no longer trust you as moderator and I'm going to keep sperging at User X forever and anyone who can hear me will hear it"?

[I'm assuming you mean that the moderator is user X and the banned user is user Y].
It depends on whether user X actually is trusted. If "everyone" trusts user X, then user Y will be "able" to "post", yes, but hardly anyone will hear him. (This is basically the current situation, except that spergs have to go on OnionFarms.)

If, on the other hand, User X is just some self-appointed janny, his ban of User Y would only be respected by those who respect him (i.e. nobody).

There are edge-cases in which a ban is controversial, sure, but I don't agree with you that it would be a serious problem. Take the bans of @SIGSEGV or @zedkissed60 as examples. If KF worked as today, except that there were a button "override mod decision; locally unban X user", and doing this allowed you to see their posts, would that be a catastrophe for the community?

I can see how it would be a mild annoyance if some threads are like Swiss cheese, where half of the posts are made by [HIDDEN USER] and the other half are quotes from [HIDDEN USER], but I don't see how it ruins the community.

The worst-case scenario that I can see is that you get a bunch of trannies signing up to the website, they decide to hide all the KF posters, and all the KF posters hide the trannies.

Is that catastrophic? If so, the two groups would form two entirely separated cliques. This is how the Internet works today - we have our sites and they have theirs, and nobody has to see what they don't like. I don't see anything wrong with that system.

As far as I know, that's also basically how the Fediverse works: there are tranny instances, there are centrist instances, there are alt-right instances, and they hide each other. That seems to work OK, doesn't it?

Or similarly - "This post is deleted" / "No, you're deleted to me".
One might argue that most people will just go with the flow, but keep in mind that anyone using an experimental distributed forum is likely to be the kind of obstreperous nerd who will customize their trust list just how they want it.

I have two points in response to this:

1.
"Charismatic leadership" is basically how forums today work! People use this site because they like (trust in) Null's moderation decisions. If they stopped trusting in those decisions, they would leave, and so Null's moderation decisions would cease to matter. There is a certain amount of inertia/wiggle room (viz. Twitter running suboptimal policies to appease advertisers), but mods do not have unlimited freedom (power) and they do not own their community. (See: the collapse of Freenode)

In a decentralized model with trust, we get most of the benefits of a centralized system (We can trust a single authority, if we want to), but we avoid the downsides.

Also consider that most moderation disagreements are not in differences of interpretations (~judicial branch), but in differences of policy (~legislative branch). To clarify: disagreements like these are relatively rare:

A: *makes post*
B: That's racist, pls ban
C: That's not racist, pls don't ban

But disagreements like these are way more common:
A: *makes post*
B: That's racist, and ban racism
C: That's racist, but don't ban racism

In such a scenario, it would be possible for users to just publicly state what their trust list policies are. I think I should clarify here that Message Trust doesn't equal Trust List Trust. So it's totally possible for me to say the following:
1. I like X's posts.
2. I don't like X's decision to assign negative trust to A, B, C, ...


2.
In the centralized model, banned users face the Hobson's choice of either to stop posting or to leave to an OnionFarms/Voat/8chan-style ghetto. In the decentralized model, they can keep posting, and be read by only the people who want to hear them.

So even if people are extremely schismatic, as you say (which is probably true), schisms would hurt the community far less than they do on centralized forums, because people with different views on moderation can coexist peacefully.

Bottom line: Even if people are autistic, it does not matter, as long as there are groups of people who are autistic in exactly the same way.

To keep discussions from going off the rails I think it's important to have one authoritative view of what is "in" and what is "out".

What do you mean by going off the rails?

If you mean non-global trust: The only real-world examples I can think of are IRC and 4chan-X for 4chan. They both support "filtering", where you hide posts/messages by certain nationalities, words, images, etc. This results in a locally inconsistent view. It is indeed slightly annoying, but I don't recall it wrecking discussions - once in a blue moon someone will write "stop responding to that guy, I have him on mute".

Am I missing something?

How complicated would a description of XenForo sound?

Thanks for the kind words. I tried to write it in detail, to convince myself, as a rough spec to refer to while implementing, and to try and convince people who have specific doubts about how something would be implemented.

I'll respond inline:

This basically sounds kinda like 'Secure Scuttlebutt', which is one of the only federated social media type thingies that actually works (rather than being a source of ever-increasing requests from the folks running sites like poa.st for donations to chuck hardware at shitty software).

Secure Scuttlebutt works, but since they don't have transitive trust (last I checked), it works less like forums and more like blogging (or Twitter). The storage part is heavily ripped from there, though, and I should add that to the README.

Fediverse also works fine, but their goal isn't to replicate Internet forums, it's to replicate Twitter. They succeed in their goal but they don't have the same goal as me.

(Fixing the money issues for Fediverse is just an engineering problem)

It is true that there is a barrier to entry for this. Hopefully, that could be handled with a nice simple installer to spin up your local node.

I do think the 'web of trust' thing is unnecessary and a means for trusted moderators to explicitly delete posts on behalf of a forum 'owner' is more important, but it sounds like you'd have concerns about users using the 'ignore' feature on regular forums too.

Well, put it this way: who is going to "own" a forum? How do we decide which moderators are trusted? Quis custodiet ipsos custodes?

As a thought experiment, It would not be a technical problem to have a URL to browse a certain board, using the imputed trust preferences of a certain user - you could even allow users to transfer around "board ownership" on the blockchain (news alert: actual use-case for blockchain found).

Spoiler: How this would work

Let's assume we use DNS (you could also use Unstoppable Domains, ENS, or any other naming service with a notion of ownership)

1. I navigate to http://localhost:1488/_dns/kiwifarms.net/b/
2. My client does a DNS lookup to kiwifarms.net and gets the TXT record for zboard. It might look something like this: zboard={"<Null's ID>": 1.0}
3. My client downloads that ID's trust list (and fetches recursively), by policies which are specified in the software (therefore global)
4. My client displays posts for something like dns.kiwifarms-net.b using the exact trust policies the owner of the DNS domain kiwifarms.net would have.

But what's the point?

Spoiler

One thing that I can see this being pretty useful for, is as a blogging spinoff - the "owner" is the guy who posts on the "blog", and he decides who gets to post. So it would work (depending on how it's implemented) either as something like a normal blogging (RSS) system (but with comments), or as Drew DeVault's public inbox.

That being said, for this to be good, you'd either have to accept very gruesome UX compromises on the blogger (browsing like a mailing list with no CSS), or implement a giant client-side templating engine.

A third path might be to expose a local JSON / unstyled HTML (AJAX) API, and allow them to host static pages that fetch the comments over the local endpoint - restricting it carefully with API tokens and whatnot.

I think this would be a very cool idea but I wouldn't rush to implement it off the bat since it's so out there, unless there is a serious need for blogging platforms all of a sudden.

If people would've trusted that user anyway, the ownership feature adds nothing, and if they wouldn't, the ownership feature is detrimental. Is there something am I missing?

As much as crypto sucks balls, if you aren't doing email verification, you probably need some sort of proof-of-work method to reduce the rate at which people can spin up accounts to troll.

Yeah -- I cover this a bit under "registration". On a technical level, the way registration works is "you exists if someone else who exists considers that you exist". But this doesn't tell us too much about the social level. The advantage of WoT is that anyone can run a registration server, and that people can "rate" them on basis of e.g. how much spam they have.

So let's say you have the following servers (all centralized):

1. PoWbot - gives epsilon trust to everyone who solves a PoW that takes ~30 minutes
2. Emailbot - gives epsilon trust to everyone who responds to a verification email
3. Gmailbot - gives epsilon trust to everyone who responds to a verification email and has an @gmail.com address
4. Phoneverifybot - gives epsilon trust to everyone who completes a SMS challenge
5. KFbot - gives epsilon trust to everyone who answers a PM on this forum

Then, people rate those servers based on the quality of the users they produce. What happens? The costlier signals will be higher quality, as decided by the free market. The trust the community places in these servers will be a function of the quality of the users they produce, which in turn will be a product of these two factors:

1. How much do I trust the underlying mode of verification (i.e. how hard is it to get a gmail/KF account)
2. How much do I trust the guy who owns is (is he secretly adding in his own Sybils too)

So, anyone who's initially trusted enough would be able to run a registration server, and that server would be rated (basically independent of its creator) on the basis of the results it produces.

Q: But isn't this basically the same as using gmail/KF outright?
A: No, because the trust is "agile". Let's say I proceed like this:

1. I register by by presenting my KF account (@hundredpercent)
2. I gain some epsilon of trust
3. The KFbot node is highly trusted, so people can see my posts, at least a little bit
4. I make normal posts and people begin to reply to my posts and trust me manually, outside of the KFbot automatic framework
5. I am banned from KF
6. KFbot stops trusting me

Result? If people still want to see my posts, even without the KF trust, we can "cut the umbilical cord" and keep living. This means that it's more robust than directly linking an account, because we can unlink it and still have a viable identity if we have a new link.

I've thought about this a lot, and I don't think it would work.

Text only would be able to make someone have a complete copy of course, so that rules out the hard drive space issue. But any splitting up of hosting to "I trust this user, I don't trust this user "would just end up fracturing and losing posts.

You would lose posts, yes. That's a feature - I don't want to see viagra spam or CSAM, and I'm more than happy to drop those.

Sometimes awful posters make good posts, even a stopped clock and all that. It would fracture things too much, and there would eventually be data loss.

Isn't that how KF works as well, too? I'm sure there's some users who are banned who would make great posts if they were allowed to come back, but nevertheless, they are banned.

One technical detail is that banned users on KF still show their old posts (but can't make new ones), whereas banned users here are 100% hidden - even their history.

I have some notion of "pinning" users to a certain version, so we could achieve a similar effect, but I think this would be more trouble than it's worth to implement honestly.

Ignoring doesn't fragment the consensus view of what is or isn't "on the forum", that's the difference. And on a regular forum, ignoring is limited to one specific form: not displaying a specific poster's content to you. You can't ignore an edit, deletion, ban, etc. And what you ignore doesn't have any bearing on others.

Technical note:

• Edits are not really supported - posts are identified by the SHA256 of their content (plus some other stuff, see schemata.py)
• The way it would work is that you either trust an user or you don't. If you trust the user, you see their posts (and also trust their deleted posts - i.e. you trust that their record of posts is true). So the only decision you can make in terms of content is "show all of their posts Y/N" - forcibly editing a user's post by means of the trust graph or something like that isn't possible.

Yes, or even just unpopular/obscure content dropping out of hosting because it's not "on anyone's list". Maybe a guy with one post in an old thread will just vanish from the record. I think this is inevitable with any sort of IPFS/Freenet-style "host what you want" strategy. It can be worked around, but doing so would defeat the purpose of having distributed hosting at all.

Is that a bad thing? Most people read threads from the past year, not what happened decades ago.

The hosting situation is comparatively better than Freenet/IPFS/BitTorrent, since trust doesn't decay unless you change it, but you can't guarantee it. However, I don't see why non-posting users would suddenly lose trust.

On the other hand, 4chan has absolutely zero retention and it seems to work fine, and Usenet is similarly non-deterministic in terms of retention; I don't think that's their biggest issue.

I mean, it would be theoretically possible to archive everything - since it's digitally signed, it'd be trustless and so the situation is better than with 4chan archives. But what's the point?

---

Thanks, everyone, for the feedback. I really appreciate it; it made me clarify some design decisions and rethink others. Personally, I think that the main issue is simply the lack of demand - since the Internet is (currently) good, there is no real reason for people to use odd decentralized software (or even Tor), but this might come to change in the future for reasons that are all too obvious.

Against this backdrop, there's only really a few possible groups of users:

1. Guys who like the idea of new technology and want to use it for the novelty - think ZFS or BSD guys
2. People who can't post anywhere else (DMCA/white nationalism/fedposters/criminal activity/acts contrary to nature). Some of these should be killed and some of them are pretty interesting to listen to. I imagine Terry Davis, being banned from all other websites on earth, might've used it, were he alive.
3. People who think they can't post anywhere else because they're deeply paranoid (schizoposters; see also: group 1)

My conclusion from all of this is that I should probably try to get a viable prototype worked out and see if it totally crashes and burns under adversarial circumstances, but that the demand for these types of softwares is quite limited at the moment.

 

[Nol 2]

Look, you can say its not blockchain all you want, if you start talking about nodes, alice and bob, and sybil resistance, Imma think its blockchain. Let me know when you throw a liquidity pair up on uniswap.

 

[⠠⠠⠅⠑⠋⠋⠁⠇⠎ ⠠⠠⠊⠎ ⠠⠠⠁ ⠠⠠⠋⠁⠛]

@hundredpercent I think the hardest bit about this is while it could work conceptually, people do tend to flock to forum communities run in the manner of a benevolent dictator- even if the rules are relatively lax and the delegation of authority to individual Leaders under the ultimate Leader is relatively diffuse, as in the case of the Farms. It's certainly a better model than oldschool Usenet however (and no less accessible to the regular Web user nowadays). That said, I'd be interested to see where you get to with it. I'm probably capable enough with Python and web FE stuff to attempt beating on things, as long as there's structure in place.

 

[Hamsteroid]

Disclaimer: I'm just a (slightly) above average tech-savvy normie and English is my second language, so a lot of the technical stuff goes probably straight over my head.

Do I understand it correctly that the personal trust levels pretty much work like this?

• Trust levels range from -1 (distrust) to +1 (trust)
• I register to the forum on NodeX –> I see every post from every user on NodeX as every trust level there equals 0 in the beginning. Visibility of any other node depends on the collective experience with / trust of NodeX's users (and bots) with that node
• I like/dislike Alice@NodeY's post -> My trust level for Alice increases/decreases by 0.1, my trust level for NodeY as well as users on Alice's trust list changes by 0.01
• I follow/mute Bob@NodeZ -> My trust level for Bob is set to +/- 1 and the my trust level for NodeZ as well as users on Bob's trust list changes by 0.1

Decentralized tech is really fascinating and I'll definitely keep an eye on this thread

 

[StyxHexenHamster1350]

Hmmm, a decentralized forum with a system of trust ratings...

That sounds a lot like Retroshare. It's an encrypted decentralized mesh, and every node operator can give trust ratings to other nodes and users. It has forums, chat, filesharing, mail, and a few other things.

And using Retroshare+Tor gives IP address anonymity. Retroshare+I2P works as well for that purpose.

 

[hundredpercent]

Disclaimer: I'm just a (slightly) above average tech-savvy normie and English is my second language, so a lot of the technical stuff goes probably straight over my head.

Do I understand it correctly that the personal trust levels pretty much work like this?

In broad strokes, yes, in specifics, not exactly.

• Trust levels range from -1 (distrust) to +1 (trust)

In principle, yes. There are two kinds of trust. Message trust (MT) goes from -1 (distrust) to 1 (trust). Trust list trust (TLT) goes from 0 (ignore) to 1 (trust). (I'm not sure if it has to, but I'm not aware of any graph algorithms that handle negative trust meaningfully)

• I register to the forum on NodeX –> I see every post from every user on NodeX as every trust level there equals 0 in the beginning. Visibility of any other node depends on the collective experience with / trust of NodeX's users (and bots) with that node

"Node" is just a fancy term for user. If your trust level (as seen from another node) is zero, nobody will see your posts (trust has to be above zero, since "0 trust" means "unknown" meaning "you can create a million accounts with 0 trust for free")

An account with zero trust is basically an account that hasn't been created/doesn't exist.

In order to start reading posts, you will have to assign at least some trust to somebody before you do anything else. If trust for all nodes is zero, you won't see any posts.

If by "register" you mean "get trusted by", that's basically right. If I only have one person/bot trusting me, it seems pretty reasonable for me to trust them, but there is no technical requirement to do so. (I could even rate them negatively if I want to)

So, something like this is more correct (numbers are purely hypothetical):

1. I create a new account/identity/node
2. I ask NodeX to trust me
3. I assign 100% trust to NodeX
4. NodeX assigns 0.1% trust to whoever signs up. Let's say NodeX has 500 users.
5. Trust of others, as seen from my node, is now something like:
   1. User 1: 0.1% trust
   2. User 2: 0.1% trust
   3. User 3: 0.1% trust
   4. User 4: ...
6. Based on how they trust each other (for example, if all of those people trust User 2 very much), my list might end up looking like this: (this is recalculated automatically)
   1. User 1: 2.8% trust
   2. User 2: 12.3% trust
   3. User 3: 0.01% trust
   4. User 4: ...
7. I now have trust list trust. I use this as weights to get the message trust of individual users. For example:
   • User 1 rates user X as +50% trust
   • User 2 rates user X as +10% trust
   • User 3 rates user X as -5% trust
   • Nobody else rates user X
   • Final trust: +2.7% (0.028 * 0.5 + 0.123 * 0.1 + 0.0001 * -0.5)
   • Basically, for users I haven't yet rated, I trust them equal to the weighted average of how others trust them (weighted by how much I, in turn, trust the rating users)

• I like/dislike Alice@NodeY's post -> My trust level for Alice increases/decreases by 0.1, my trust level for NodeY as well as users on Alice's trust list changes by 0.01

In principle, yes. Those numbers could be anything, but basically yeah. Assuming you increase both Trust List and Message Trust by 0.1, and Alice trusts all the users on her trust list exactly +0.1, then the increase is +0.01. Depending on how strongly she (dis)trusts them, the change is proportionally weaker.

Note that Alice is not "on NodeY", she just happens to be trusted by them. Trust doesn't propagate upwards ("I trust X who is trusted by Y, so I also trust Y"), only downwards ("I trust X who trusts Z, so I trust Z")

• I follow/mute Bob@NodeZ -> My trust level for Bob is set to +/- 1 and the my trust level for NodeZ as well as users on Bob's trust list changes by 0.1

For the positive side, basically yes. But if you give them a strongly negative trust, this just means you ignore their trust ratings, TLT can't go negative. (Otherwise, I could create a million accounts that just spammed gore or whatever, add 100% trust to people I dislike, and cause problems.)

Decentralized tech is really fascinating and I'll definitely keep an eye on this thread

Thanks, glad to hear it.

 

[hundredpercent]

Hmmm, a decentralized forum with a system of trust ratings...

That sounds a lot like Retroshare. It's an encrypted decentralized mesh, and every node operator can give trust ratings to other nodes and users. It has forums, chat, filesharing, mail, and a few other things.

And using Retroshare+Tor gives IP address anonymity. Retroshare+I2P works as well for that purpose.

Retroshare is very interesting, but its friend-of-a-friend design limits scalability. Assuming authenticated, public topics, the official blog (2013) says:

Localisation is a characteristic seen primarily in authenticated Forum topics. It means that topic messages are only received that originate from a few hops away on one’s Retroshare network. This serves to keep the discussion relevant to a peer’s circle of friends, and enhance your privacy. Localisation is caused by peers not distributing messages which they cannot authenticate (i.e they don’t have the author’s PGP Key). Typically, these are messages from authors 3 or more hops away.

In plain English: the visibility of your posts is very low, since there is no adequate spam filter mechanism.

 

[Kosher Dill]

"Charismatic leadership" is basically how forums today work! People use this site because they like (trust in) Null's moderation decisions.

I think this gets to the heart of where our views differ. The way I see it, people use this site because they want a community where everyone is operating on the same "rules of the road" that they like. It's more than just having your personal trust in the head moderator set to 100%.

I mean, it would be theoretically possible to archive everything - since it's digitally signed, it'd be trustless and so the situation is better than with 4chan archives. But what's the point?

I think having an archive is exactly the point, especially at a place like this where archive-diving is the national sport. If we want a non-archived discussion place there are plenty of bad solutions already available, as you mentioned.

At any rate, it's easy to create something that looks like a forum if you're willing to hand-wave away the tricky parts. But I think those tricky parts are actually key aspects of the experience.

In fact, here's a question: what do you consider the essential features of a forum to be?

 

[Rusty Metal Skull Gun]

I agree with @Ridley: while interesting, this is over-engineered. No-one's going to download and run a Python program just to shitpost about trannies. Plus it effectively excludes phoneposters (this may not be a bad thing).

You're also fixing problems the Farms doesn't really suffer from, namely spam and poor moderation.

I've been giving some thought to a similar issue, i.e. how to make a troon-resistant KF. I'm wondering if it's not insane to do away with the database entirely, and store all posts in Git, and all rendering etc. done by JS in the browser. This would mean:

* It's trivial for anyone to take a local copy of the forums ("git pull"), and trivial to restore a backup
* Reading threads just involves reading static files, which is as fast as things can possibly be. There's no reason why the server should waste resources in e.g. choosing the random_text.txt message, when this can be off-loaded to the user. There'd still be a script to handle posting and anything else that changes the repo.
* Because everything has a Git hash, you can use this as an HTTP ETag, make everything very cachable.
* You can use service worker shenanigans to download a whole thread to read offline, and so on
* If there's ever a DDOS attack, Josh can shift the whole thing into read-only mode, rather than the whole thing being forced online. Look at how the static "sorry" pages are usually fine, even when the site is being hammered. PHP and DBs are both resource-intensive.
* If the whole thing collapses, someone can just take a recent snapshot of the repo and declare that to be the new Kiwi Farms

Obvious problems include how you handle searching, and what you do about stuff like passwords.

I've never seen anybody use Git as a database, so I don't know if this is retarded or not.

 

[888Flux]

Decentralization needs to happen on the infrastructure level. There are many alternative protocols and applications such as Tor, Yggdrasil, and I2P but they rely on the same infrastructure other networks do. The infrastructure that makes up the backbone of the Internet is complex and fragile. Ultimately the problem is not on the application layer but on the physical one.

Even in a peer-to-peer setting like the system you propose the packets are being sent down the same carrier as any other packet on the Internet. It's being sent through a network that is far beyond your realm of control and consists of thousands of interconnected moving parts each with their own point of failure. At any time this connection can be disrupted or shut off entirely.

The problem should be solved by creating a network of physical infrastructure that spans multiple carriers. A physical mesh network system that integrates both existing network overlay protocols and new physical infrastructure. This is possible and there are many realistic ways to achieve it. But it is the only long term solution to the problem at hand. The Internet will only continue to regress into a walled garden content delivery platform much like TV was. Kiwifarms represents a microcosm of the coming end of network freedom. Regardless of who you are, your beliefs or your worldview, the full scale attack on a website like Kiwifarms should be alarming. Much of the push to deplatform the site has come from personal interests. Imagine what a deplatforming campaign wil look like in the future when it comes from corporate or government interests.