Programming thread

[Safir]

Spoiler: Help me i'm trying to make a minigame aaaaa

Spoiler: how to add click sounds for main menu buttons and options buttons?

Spoiler: Help with action Jump to label

Spoiler: How do you make the main menu music stop

Spoiler: So the character.png isn't showing but the blank gray one is.

Spoiler: Jumps won't work

Spoiler: I dont really understand why it keep showing me this error

Spoiler: I need to know what should I name the .ttf file

Spoiler: Transferring data from browser app to Ren'py

Spoiler: No se que hacer

Spoiler: WHAT THE FUCK IS THIS FUCKING SHIT?!

Notice a trend?

 

[${Sandy}]

What really sets CoE apart to my mind is the amount of real historical / mythological detail in the different factions. Necromancers make use of hands of glory which are a real-life would-be magical item. The Priest King faction, like I said, is based on Mesoamerican lore and plays fast and loose with some details but overall there's decent correspondence with actual Aztec or Mayan beliefs. There's another faction called Cloud Lords whose names sounded to me kind of Indian or Persian and I found out they were Avestan names, which is an ancient Persian language. As it turns out, there are actual Persian legends about winged people with questionable morals, like in the game. There's a Demonologist faction where, if you try hard enough, you can even summon characters like in Dante's Inferno such as Geryon, Monster of Fraud. There are Lovecraftian entities too from the Void, made use of by the High Cultist faction, as if all of the other things weren't enough. You can bring gods into the world and kill opponent gods too. A smaller but still funny detail is this game is made by Scandinavians and moose, bears, deer and even fish-people who can be killed and smoked to feed entire villages are a constant low-level menace. These are the sorts of things that make CoE really worth playing from the days of the Atari ST up to the days of Steam in the current year.

Sounds similar to a game called Dominions but I haven't played either.

 

[Safir]

Sounds similar to a game called Dominions but I haven't played either.

Same devs. Conquest of Elysium is, roughly speaking, single-player Dominions.

 

[${Sandy}]

Ive done several coding projects over the past year but the one I've recently taken on has a wider scope than anything I've done before. Im determined to see it through and have something impressive for my portfolio and I decided I need to come up with a game plan.

Can somebody tell me how you would recommend planning out a large project for coding? I have a lot of ideas and I was thinking about taking the time to organize them and plan them out before I write any more code so I can work more efficiently.

 

[ADHD Mate]

Ive done several coding projects over the past year but the one I've recently taken on has a wider scope than anything I've done before. Im determined to see it through and have something impressive for my portfolio and I decided I need to come up with a game plan.

Can somebody tell me how you would recommend planning out a large project for coding? I have a lot of ideas and I was thinking about taking the time to organize them and plan them out before I write any more code so I can work more efficiently.

Anytime I tried to plan for something I had no experience doing it was wasted effort.
I would recommend writing list of high level design goals, choosing one of it and splitting it in few tasks. Not too many though, even one will probably be enough.
If you notice some improvements/issues then add it to your backlog.

Other than that, I don't think there is much of a replacement for trial and error.
Also, don't be afraid to trash your code, that's proof you are learning.

 

[306h4Ge5eJUJ]

I would recommend writing list of high level design goals, choosing one of it and splitting it in few tasks. Not too many though, even one will probably be enough.

To expand on this a bit, it helps to look for things that could reasonably be handled by a third-party library, even if you intend to write it yourself and don't intend to reuse the code elsewhere. That way you have a smaller, almost self-contained subproject you can start and experiment with more quickly. It also often encourages a cleaner design because there's less temptation towards dirty hacks that break the abstraction of your pretend library.

This is all rather generic advice of course since the question is generic.

 

[ADHD Mate]

It's hurtful how true it is nowadays.
How many micro-services in my current jobs are just repackaged OSS.
Trying to debug anything is often impossible as people responsible for those repackages have no idea what their services even do.
So it's just endless back and forth of Jira tickets with some hacky fixes..

While having some basic understanding of core concepts of given tech suddenly gets you called an expert.
Stackoverflow and incompetency memes ruined a whole generations of programmers.

Also having to deal with all great internal libraries that just obfuscate what needs to be done, yet does not provide any standardization at the end is great.

At least the pay is great and I have time to do other things during work-hours.

 

[Hall of Cost Payer]

Can somebody tell me how you would recommend planning out a large project for coding? I have a lot of ideas and I was thinking about taking the time to organize them and plan them out before I write any more code so I can work more efficiently.

One of the most important approaches is to get something running quickly. Don't build abstractions and classes and whatever you have without running compile, having a small program that runs even if it does nothing yet is the perfect starting point to change slowly into what you want it to do.

This ties in perfectly with

it helps to look for things that could reasonably be handled by a third-party library, even if you intend to write it yourself and don't intend to reuse the code elsewhere.

I also agree on the trashing code and rewriting it. Planning a big project is one thing, after starting to program it, it is very likely you will see a better way to do it. I think it's better, after reaching a satisfying milestone first, to rewrite it with the lessons learned. Nothing is better for your own architecture than to learn from it.

Depending on the type of project, maybe try to find similar open source projects online and learn from them. For example if you were writing a web server, finding an open source project could show you how the Controller, Service, Repository may be separated. Or if you wrote a game, looking at open source games may show how they did Entities, enemies, input handling etc. But as the saying goes only take inspiration and don't copy without understanding

 

[SleeplessEcho]

Can I get a C++ tip from any passing 400-pound C whales?

In Windows, you can say "wait for a certain object to become signaled, or until X milliseconds pass", by simply: WaitForSingleObject(handle, msecWait);
Which you've been able to do since approximately the Jurassic period.

What's the best way of doing the same in modern C++ using the std::thread library?
The most obvious analog, using a std::condition_variable, seems like a hack.
(example: https://en.cppreference.com/w/cpp/thread/stop_callback)
I've also seen suggestions of using a std::future that you can either wait for or have it return early if "signaled".

https://en.cppreference.com/w/cpp/thread/condition_variable/wait_until

This looks like it'll do what you want it to. I don't see how using std::condition_variable is a hack; from my Linux-programmer perspective condvars are the first thing that come to mind.

 

[analrapist]

wait for a certain object to become signaled, or until X milliseconds pass

This is the Windows flavor of OS metaphor.

wait for an fd ready to read, or until X milliseconds pass

This is the UNIX flavor of OS metaphor: poll(), select(), et cetera. This does not use the std:: namespace as you requested, but perhaps you'd rather use OS primitives instead of language primitives. See https://daniel.haxx.se/docs/poll-vs-select.html for more.

 

[Rusty Metal Skull Gun]

So apparently std::condition_variable is implemented by a "futex", which is a sort of mutex that only does a syscall if it absolutely needs to. Even if it's trickier to use, it should be faster than doing the raw syscalls poll/epoll/select (on Linux) or WaitForSingleObject (on Windows).

 

[analrapist]

faster than doing the raw syscalls poll/epoll/select (on Linux)

Yes! These old APIs are compatible but slow. io_uring ( https://en.wikipedia.org/wiki/Io_uring ) is liable to be the most performant from a syscall angle, and if you can avoid the syscalls (eg. futex), that's yet faster.

 

[macrodegenerate]

Ive done several coding projects over the past year but the one I've recently taken on has a wider scope than anything I've done before. Im determined to see it through and have something impressive for my portfolio and I decided I need to come up with a game plan.

Can somebody tell me how you would recommend planning out a large project for coding? I have a lot of ideas and I was thinking about taking the time to organize them and plan them out before I write any more code so I can work more efficiently.

My advice is to sit down and figure out what you need to make a minimum viable product. Then determine how much of what you need can be provided by OSS projects. Think for a day or two and figure out how to sow things together, and then start. You may also want to look into your cloud provider, and figure them out first.

 

[${Sandy}]

My advice is to sit down and figure out what you need to make a minimum viable product. Then determine how much of what you need can be provided by OSS projects. Think for a day or two and figure out how to sow things together, and then start. You may also want to look into your cloud provider, and figure them out first.

My current project is C# and doesnt involve internet but Im planning on upgrading my computer soon and setting up a personal website and back end server as my next big project.

 

[Marvin]

So my day job seems to be in a financial crunch and we're all on indefinite furlough (if it ever ends) so I'm starting to look for alternative, probably permanent employment elsewhere.

The job hunt sucks.

Though thankfully, the recruiters have started to pick up more now that the election's over.

---

In the meanwhile... I'm back to working on my little personal project. (First post, second post, third post)

So where we left off, I have a daemon that manages media streams. (all this language is euphemistic, I'm not actually working with media streams; just in case this project goes anywhere, I don't want to be doxed from my posts here)

To recap, the idea is that you feed it a config file specifying sources, sinks, and tasks. Each task reads from its sources and writes to its sinks in an infinite loop. Runs as fast as possible.

Multiprocessing ()​

Each task gets run in its own thread. I've never been comfortable with threads, I've always preferred fork() and IPC as alternatives, because I'm super paranoid about fucking up mutexes and deadlocks and things like that. But I've dipped my toe in.

Theoretically each task runs fairly independently. Each of them have a [URL='https://ocaml.org/p/domainslib/latest/doc/Domainslib/Chan/index.html']Chan.t[/URL] where basic commands like Exit or BlockSource can be fed into the task. The tasks only poll, they don't block, on the command channel.

So the main loop for a task looks roughly like this:

let run task =
  let running = ref true in

  let rec handle_events () =
    let handle_event event =
      (* handle the event here *)
    in
    match Chan.recv_poll t.ingress_events with
    | Some event ->
      handle_event event;
      handle_events ()
    | None ->
      ()
  in

  let stream_single_pass () =
    (* read something from task.source and then write it to task.sink *)
    (* btw this function does block for a period of time if nothing's available *)
  in

  while Ref.(running.contents) do
    handle_events ();
    stream_single_pass ()
  done;

  ()

So the function has an infinite loop and it polls the event channel without stopping, but does poll and stop on the actual stream function. Theoretically an attacker could starve the stream function by spamming events, but those events are only from my code, not open to the public.

Also, btw, a little rant about references...

So Ocaml is a functional language. All its variables are immutable. But unlike, say, Haskell, it isn't purely functional.

I do think functional programming will improve your code, it makes things more debuggable and readable, and harder to fuck up.

But some algorithms simply are easier to reason about with side effects. Not all, and I do encourage people to try and reimagine their algorithms in a functional way. Functional programming, where it works, does pay dividends.

But it's not universal. Again, I'm not a purist.

So one simple way to add side effects in Ocaml is to use references from the standard library. They're a container holding a single cell that you can read and write to.

Except the read syntax is absolute ass.

Here's what a side effecting reference use looks like:

utop # let my_mutable_variable = ref 3;;
val my_mutable_variable : int ref = {contents = 3}

utop # !my_mutable_variable;;
- : int = 3

utop # my_mutable_variable := 4;;
- : unit = ()

utop # !my_mutable_variable;;
- : int = 4

Yes, the default operator to fetch a value from a mutable reference cell is !. The assignment operator isn't so bad, :=, but using ! is super confusing if you're using the reference cell for a boolean variable.

Ocaml is a great language (the whole ML family is, really, and Ocaml is really just the one that took off), but goddamn, the French get up to some goofy shit sometimes.

So instead of the illegible !, I just looked at how ref is implemented in the standard library. It's actually a structure with a single mutable element called contents.

So Ref.(running.contents) instead. Far more readable.

Commands and Monitoring​

Now part of my thinking was that this daemon would be fast but dumb. I will put some pretty basic decision making primitives in the daemon itself (think "if above X bytes, redirect frame to other sink"), but leave most of the hard work of programming those primitives and analyzing data to outside processes.

See I've got a friend whose strong suit isn't coding, but he's really good with machine learning algorithms and the relevant math. He's a very typical data science type, who works with python.

I always thought that I could build the raw material, and feed data to something he's written in python, and in turn, he could feed commands back to my daemon.

So the model is, raw monitoring data is streamed out of the tasks, out of the process to any listening programs that might be written in whatever language.

And in turn, those outside listening programs can send commands to the daemon to control its behavior.

Commands​

Commands aren't particularly difficult.

I could probably get by with a http listener that takes POST'd JSON payloads.

But just to save work, I'll probably duplicate whatever I use for Monitoring.

Monitoring​

This is where things get trickier.

I can see myself streaming a few different types of data. Some of it might be simple data structures. Stats about various things. Key value maps, pointing to numbers, or maybe further nested structures. Something that JSON would probably work for.

But... I also want to be able to, when appropriate, just memcpy things from the working data into a buffer and fire it off.

If I'm dealing with media frames, I can save a lot of time by just memcpy'ing the headers of the frames and streaming them out of the process to be analyzed.

So the format I choose should probably be binary friendly. That rules out JSON.

Also, btw, there's a small issue of aggregating the data streams from each individual task thread to one big firehose for outside consumption.

That is, if I've got 5 tasks, running in 5 threads, in one process, I want to provide a unified socket to the outside world to monitor. Not five sockets.

So I'm going to want to have some sort of funnel thread that condenses the different task monitoring streams into one output stream.

I could finagle some tcp connections (with the option for unix sockets when available), but then I ran into zmq. It's a really nice wrapper around common networking patterns, and uses very a familiar (zmq_) socket, connect, bind, send, recv paradigm, but under the hood it has a lot of convenience code to handle common issues (disconnects and 1-to-many arrangements and things of that nature).

That doesn't really change much about the meat of my work, but it's nice. I'd definitely give the guide a read. In particular, this is thought provoking:

The point is that while the Internet offers the potential of massively connected code, the reality is that this is out of reach for most of us, and so large interesting problems (in health, education, economics, transport, and so on) remain unsolved because there is no way to connect the code, and thus no way to connect the brains that could work together to solve these problems.

There have been many attempts to solve the challenge of connected code. There are thousands of IETF specifications, each solving part of the puzzle. For application developers, HTTP is perhaps the one solution to have been simple enough to work, but it arguably makes the problem worse by encouraging developers and architects to think in terms of big servers and thin, stupid clients.

So today people are still connecting applications using raw UDP and TCP, proprietary protocols, HTTP, and Websockets. It remains painful, slow, hard to scale, and essentially centralized. Distributed P2P architectures are mostly for play, not work. How many applications use Skype or Bittorrent to exchange data?

Which brings us back to the science of programming. To fix the world, we needed to do two things. One, to solve the general problem of “how to connect any code to any code, anywhere”. Two, to wrap that up in the simplest possible building blocks that people could understand and use easily.

Anyway, it's not a huge deal, I'm more worried about choosing a binary format.

Binary formats​

So I wanted something to handle a bit of nesting, with support for binary data, that's quick to serialize, and ideally, is supported among other languages/runtimes.

Let's look at what I tried and discounted first:

1. bin_prot https://github.com/janestreet/bin_prot

Bin_prot is a binary serialization library, plus a language syntax extension that automatically generates serializations for your Ocaml types. You can have it generate serializations for parts of your code, and manually implement specific serializations for other parts of your code, and it all just works together fairly easily.

This one I kinda knew was a non-starter because I'd have to reimplement its protocol by hand in Python or Go or whatever else I wanted to support.

But I wanted to give it a try, just because I've run into it before and it seemed neat.

And indeed, it was pretty neat. The way it works is that there's a tag you add onto the end of a type and it generates serialization functions:

open Core

type stream = {
    index : int
  ; name : string
} [@@deriving bin_io]

type frame_metadata = {
    src : stream
  ; dst : stream
  ; payload_length : int
} [@@deriving bin_io]

This generates a module with a signature something like this:

Spoiler: long gibberish signature

type stream = { index : int; name : string; }
    val bin_shape_stream : Bin_shape.t
    val bin_size_stream : stream Bin_prot.Size.sizer
    val bin_write_stream : stream Bin_prot.Write.writer
    val bin_writer_stream : stream Bin_prot.Type_class.writer0
    val __bin_read_stream__ : (int -> stream) Bin_prot.Read.reader
    val bin_read_stream : stream Bin_prot.Read.reader
    val bin_reader_stream : stream Bin_prot.Type_class.reader0
    val bin_stream : stream Bin_prot.Type_class.t0
    type frame_metadata = { src : stream; dst : stream; payload_length : int; }
    val bin_shape_frame_metadata : Bin_shape.t
    val bin_size_frame_metadata : frame_metadata Bin_prot.Size.sizer
    val bin_write_frame_metadata : frame_metadata Bin_prot.Write.writer
    val bin_writer_frame_metadata : frame_metadata Bin_prot.Type_class.writer0
    val __bin_read_frame_metadata__ :
      (int -> frame_metadata) Bin_prot.Read.reader
    val bin_read_frame_metadata : frame_metadata Bin_prot.Read.reader
    val bin_reader_frame_metadata : frame_metadata Bin_prot.Type_class.reader0
    val bin_frame_metadata : frame_metadata Bin_prot.Type_class.t0

It's hard to read, but the important part is that it generates the functions bin_read_frame_metadata, bin_write_frame_metadata and a bin_size_frame_metadata.

The writer function mentioned above takes a buffer, a position in the buffer to write to, and the value to write. The reader takes the same and returns the parsed item. The sizer returns the size of the buffer you'd need to allocate.

Pretty straightforward.

Now here's where we might need to write our own serializer. Let's say we want to add a Uuid to our frames.

open Core

type stream = {
    index : int
  ; name : string
} [@@deriving bin_io]

type frame_metadata = {
    uuid : Uuidm.t
  ; src : stream
  ; dst : stream
  ; payload_length : int
} [@@deriving bin_io]

I get:

# #mod_use "/tmp/uuid_sample.ml";;
File "/tmp/uuid_sample.ml", line 11, characters 11-18:
11 |     uuid : Uuidm.t
                ^^^^^^^
Error: Unbound value Uuidm.bin_shape_t

So what I would do is override the Uuidm library and add in custom serialization functions:

module Uuidm = struct

  include Uuidm

  let bin_shape_t = bin_shape_bytes

  let bin_size_t uuidm =
    Uuidm.to_binary_string uuidm
    |> Bytes.of_string
    |> bin_size_bytes

  let bin_write_t buf ~pos uuidm =
    Uuidm.to_binary_string uuidm
    |> Bytes.of_string
    |> bin_write_bytes buf ~pos

  let bin_read_t buf ~pos_ref =
    bin_read_bytes buf ~pos_ref
    |> Bytes.to_string
    |> Uuidm.of_binary_string
    |> Option.value_exn

end

Basically I used the Uuidm.to_binary_string function (produces something that looks like "²³G\014\002K\000²\021/_x·5R") and then passed it to the built-in byte readers+writers.

So then the frame_metadata bin_io expansion now works.

Type based macros like these are pretty common in Ocaml, like for JSON or whatever, so it's not unusual for me to override external library type signatures like that. Most of my projects end up having a project-wide overlay.ml file that's filled with various useful extensions I've added into the various libraries I used.

In fact, that's what open Core is. The Core library is basically a private overlay that adds a lot of useful features to the standard library. Tbf, the standard Ocaml library is kinda limp. It was only really meant to bootstrap the compiler, so for most practical work, you'll want to incorporate something like Core.

Anyway, I really liked bin_io and I really wish I could've used it, but alas, I'm too lazy to port those serialization functions to Python. It'd be brittle and a pain in the ass. Oh well.

Onto something else...

2. ppx_protocol_conv - https://github.com/andersfugmann/ppx_protocol_conv

This is another type expander. It is more of a framework for type expanders, with a few different formats it supports.

And in particular, it has a MessagePack type expander. Up until this point, I had never heard of MessagePack. Apparently it's like some sort of binary JSON, which sounds promising.

Python code would be able to deserialize it and it would fill in the field names without me having to manually write special code to do it.

But unfortunately, ppx_protocol_conv isn't maintained particularly well. In particular, it's not building properly on up-to-date Ocaml version.

That sort of thing is a very real downside to using exotic, dumb languages. (Even if I really like my exotic dumb languages.)

There's genuine market value to using a less powerful, but more "commercial" language in production at a job. I have worked jobs where I was permitted to use more exotic tools like Ocaml, and I was able to manage the limitations well, but that was something I had to prove and demonstrate to our CTO (small company). Not something that could be just assumed.

The opposite though, jobs where the technology is wholly pajeet-tier and terribly boring will wear on your soul and wear your enjoyment of coding as a hobby in general. You'll get home and never want to look at another computer again if you never have fun. But ideally there's a happy medium.

Anyway, finally...

3. protobuf - https://github.com/mransan/ocaml-protoc

This is the one I ended up settling on. Protobuf is sorta annoying, but honestly it's probably the best fit for my needs.

Protobuf is a special language for specifying binary formats. That is, you specify the fields of various structures semantically (ie in terms of "I need an int, then a string, then two floats") and you run the protobuf file through a special compiler to generate parsing and serializing code for your specific language.

I first encountered it years ago working on code that would run in a data center.

To adapt my bin_io example above, this is approximately what I'd use:

package media_lib.messages;

syntax = "proto3";

message Stream {
  int32 index = 1;
  string name = 2;
}

message FrameMetadata {
  Stream src = 1;
  Stream dst = 2;
  int32 payload_length = 3;
}

The downsides aren't huge, but they're there. Like there's just the general headache of hooking an external tool into your build system. Doable, but annoying.

And then the generated code isn't really very nice. With bin_io, I could write much nicer, much more Ocaml'y code first, and then let the plugin handle it or manually write serialization code when necessary.

The protobuf generated code isn't terrible, but I do have it segregated into a separate module and just import it and wrap it up so it's nicer to use within the rest of the code.

So yeah, protobuf it is.

---

Currently I have the protobuf compiler hooked into the build system (I'm using dune). I have written code to read/write it from nicer written ocaml structures.

And I have a test ready to go. The test I have written is watching the monitor stream (streaming protobuf messages over zmq, as stated above) and when a certain media frame comes through, it sends a command (also protobuf encoded, sent over zmq) to block that media stream.

And the test will succeed if only one media frame from that stream comes through.

Most of the monitor function bodies are currently empty, they just run failwith "todo finish this function".

So now that I have everything compiling, I'm just going to keep on filling in function bodies and running the test above until it passes.

 

[emerge]

Yes

No.

 

[Safir]

In the meanwhile... I'm back to working on my little personal project. (First post, second post, third post)

No homo but this post is great literature, even though (because?) I understand like two words per sentence. In a hundred years, a historical fantasy novelist is going to copypaste it wholesale into his novel entitled "Seventeen Ways to Defend a Firewalled Server".

 

[Marvin]

No homo but this post is great literature, even though (because?) I understand like two words per sentence. In a hundred years, a historical fantasy novelist is going to copypaste it wholesale into his novel entitled "Seventeen Ways to Defend a Firewalled Server".

I can't wait for the literary analysis. The first phd paper about it in Autistic English Literature will be off the chain.

 

[We Are The Witches]

I'm a noob compared to the people in here, would you agree with ChatGPT on this? It relates to a hypothetical where in Python you'd only have one method (like split) with an additional argument in order to determine where should it start from, instead of having the additional method rsplit.

So instead of using rsplit(), why not have something like split(from_start=False)? Something like that.
Disregard the backwards compatibility argument, and also, I understand the answer, but I don't see the big deal with having it as in the hypothetical. Is there something else as a reason, or just this?

Spoiler: Answer

If someone decides to answer, thanks in advance.

 

[Sperg Coalition]

-snip-

I can agree on the fact that having two separate functions for splitting strings starting from different directions is better for Python specifically: consider that people that program in Python are either beginner programmers or experts that need some quick scripts where clarity is preferred instead of "clever code".
One criticism that I have in mind is that if you really need to separate these two splitting functions, you should name them appropriately, we are not programming in C in the 80's anymore: split_from_left and split_from_right let you immediately understand what they do instead of having split and rsplit.