Sneedforo

  • 🐕 I am attempting to get the site runnning as fast as possible. If you are experiencing slow page load times, please report it.
Concentrate Juice said:
Do you have a roadmap or plan set out for long-term development? It would be useful to have a list of things to be completed whereupon the forum could be considered "production ready".
Click to expand...
It has to be very feature complete to be production ready because there's going to need to be an entire database import and anything not imported is lost.
 
  • Dumb
  • Like
  • Informative
Reactions: Napoleon III, The Pulverizer, Catgirl Fornicator and 1 other person
1715031488510.png

yes, over the weekend i wrote an incredible amount. I moved all the database code into a more streamline lite driver. I refactored the ways templating was done into an early stage of views, which is more organized. It's also very fast.

1715031841300.png


The code is organized now so that general concepts are stored as Collections which can be split off into what I call Collates as needed. The collate data are almost entirely references, so there's no movement or copying of data, just references being organized. That struct stores a ton of information but is so small it can cache in the cpu. Very large structs have to be read over multiple CPU cycles and cannot be cached.

The technology I'm using (Scylla + Rust) is very bleeding edge. There's not a lot of convenience. I'm writing stuff I'd never have tried to write a few years ago. But I have a lot of the design patterns memorized because I've written similar web software a bunch of times now and I know how I want it done.

Right now I'm working on making the act of creating a new post fit into the design. Since it's a PUT, I have to expand my work more.
 
  • Like
  • Winner
  • Feels
Reactions: Dixieland Buckaroo, 3MMA, DashaKaban and 13 others
1715042802770.png

After a lot of work, posting threads now also communicates using new repository code.

One of the most difficult things about the entire project is pagination. The little post counters on the right are unironically the most complex system in the database design. Scylla is blazing fast and can be distributed across multiple continents, which is why I picked it. However, as a downside, it HATES sequential values and it HATES atomic locking. It flat out doesn't exist in its design: you cannot insert a bunch of rows from all over the world to databases on different continents and know for sure that posts are going to be in order 1, 2, 3, because it just doesn't exist.

However, unlike modern chatrooms and other designs where order is more flexible, forums are kinda old and rigid. A thread has 20 posts per page. A node has 100 threads per page. It's set in stone.

Now while it's pretty easy to figure out that you have 10 posts in a thread and that the next one is going to be in position 10, forums are even more dumb than that and soft-deleted stuff doesn't push the positions. If I delete a bunch of spam, you'd not want to see 3 posts per page and 0 posts the next, etc. That means positions are both very inflexible but also necessarily need to change.

Since I can't be sure that a thread is going to order its posts correctly, the next best thing I can do is check to make sure that the thread's MAX(position) field is equal to its atomic locking thread_replies value. If it doesn't match, I know that time and space tore apart a little, and I can flag the thread as needing a repair for its ordering. This can happen automatically, and it won't be hard to set up, because I've already figured out how to do internal jobs.
 
  • Like
  • Informative
  • Winner
Reactions: DashaKaban, Pee Cola, teriyakiburns and 6 others
Null said:
Since I can't be sure that a thread is going to order its posts correctly, the next best thing I can do is check to make sure that the thread's MAX(position) field is equal to its atomic locking thread_replies value. If it doesn't match, I know that time and space tore apart a little, and I can flag the thread as needing a repair for its ordering. This can happen automatically, and it won't be hard to set up, because I've already figured out how to do internal jobs.
Click to expand...
Are you going to do the repairing of the order of posts using something like a recursive function that assigns (n-1) to a posts value in the database until it hits the recursive limit?
 
Null said:
So now I'm kind of lost at what to do. There's still a lot to be done but I want to stay mostly in Rust. Getting attachments actually attached to posts may require doing work with JavaScript, which I don't want to do right now. I have not really put much thought into how I want to do the frontend at all.
Click to expand...
Are you taking the approach of having most of the page pre-rendered on the server side or are you going Web App and having the backend be mainly a REST API with as much of the layout as possible moved to the client? I saw you talking about ordering of posts so it felt more like the former. But are you / will you be using a framework like VueJS or React? VueJS is typescript top to bottom and I found it more intuitive than React, fwiw.

EDIT: Also, what sort of metrics are you going to use for comparing your replacement to Xenforo? Got to be able to say "page load is 87x faster" or "DB indexes are < blah GB now". :)
 
1715109881428.png

Attachment thumbnailing now displays correctly. This the end result of the super-lean collate system I was discussing, and how it accepts a ton of organized db data and then constructs the associations entirely by reference (i.e. borrows, rust's pointers), and these collates can be cached by the cpu because they're so small. The full thread page takes 6ms to load and is measured in nanos from when the request is received vs. when the footer of the template is rendered - the last thing. I think that this will get below 5ms when I can put each post's rendering into a different thread, right now this is single-threaded for debugging.

The only thing missing with my thumbnailing right now is animated webp for gifs, and it needs to populate media information. I need to store trivial things like image dimensions. I've built this so that the attachment system will be extremely robust, and attachments will have srcset attributes for high quality thumbnailing across pixel densities.

1715110186384.png
 
  • Like
  • Winner
  • Informative
Reactions: SoyRageEngaged, DashaKaban, Pee Cola and 10 others
Will you try to mimic current XenForo UI?
 
Would the new software use real-time updates for threads and posts? No need to refresh/load new posts and you can instead see them posted immediately? Seeing Rust/Scylla leads me to believe so but dunno.

Small thing, but would there be markdown support for post formatting? GitHub and Discord both use MD/MD-like syntax for their formatting so it would make it easier to copy text from other sources.
 
Awwwww. Saw a thread bump and thought there'd been some news.

Do you have any public milestones for this? Would love to keep up to date with how this is going. Just on a technical level aside from it being the Farms, I'm fascinated by a JS-less, low-level backend web forum. This thing should go like shit off a shovel.
 
General question for tech retards: if I wanted to program Sneedchat for Sneedforo, with the consideration that making it more widely accessible outside of the website itself would benefit its activity and popularity, what protocol would you use for this?

@dumbledore suggests IRC, which is also used by Twitch chat.

This is a very broad RFC. I'm early in development still but it would be really cool if people could be in chat using their favorite app. But I personally literally just use the Kiwi Farms, Telegram, Signal, and very limited XMPP, so I don't have a high level of input on this.


Edit: I am also curious about, but lack information on, how an m3u8 playlist of smaller mp4 files is better for video playback vs. keeping original or remuxed files and using the slice / byterange headers.
 
Last edited:
  • Like
  • Thunk-Provoking
Reactions: Vecr, 3MMA, Avenue and 1 other person
IRC is the simplest protocol out there so go with that first. Just remember to use a token for logging in because otherwise your ugandan VPS providers can steal passwords. A Telegram bot would also be nice.
Null said:
Edit: I am also curious about, but lack information on, how an m3u8 playlist of smaller mp4 files is better for video playback vs. keeping original or remuxed files and using the slice / byterange headers.
Click to expand...
Seeking large files using byte ranges often takes multiple requests because the player doesn't know which byte offset lines up with the desired timestamp, it basically has to do a binary search.
m3u playlists contain the durations of all segments which solves this issue.
Another reason might be that some proxies only cache whole files and not ranges.
 
You must log in or register to reply here.
Back