Programming thread

  • 🔎 If you see an alert for analytics.usips.org, this is a self-hosted system I am running as a test. It is GDPR compliant and collects no invasive information.
  • Want to keep track of this thread?
    Accounts can bookmark posts, watch threads for updates, and jump back to where you stopped reading.
    Create account
I don't suppose they've documented this process anywhere and/or open-sourced any related tools, have they? I'd be interested to see if they've come up with something genuinely novel here or if they're just being fucking morons again.

Lunchtime sneeding so forgive me being glib:

github.com/bazelbuild/remote-apis <- here
Everything is ninja compatible. Blaze -> Bazel -> Goma -> Reclient -> Siso is the newest one. spec is open so I think the current pile of things that do it is Buildbarn, BuildBuddy, EngFlow, NativeLink, and Bazel Buildfarm

I will have time to explain more later tonight but I might also have more sherry in me soooo
 
types are great and bags of properties rule but OOP is shit
Objects are amazing, OOP is not.

That's one of the core principles I program by.

Also, better to use a fairly limited amount of different shapes of data, with many functions that operate on those, is so much nicer to work with than an OOP system where you have a bespoke shape of data for just a few functions each.
 
Dogshit is overstating it. It is much better than what was typically used before.
Nah, "dogshit" is accurate. Words of adequate offensiveness have yet to be devised in English to describe git's competition, though. As I said, it's better than the rest, but that doesn't stop it being dogshit.

I've been reading about pg-promise and pg-listen for a full-fledged messaging implementation, this is absolutely amazing.
Very cool stuff for sure. Firebase can fuck right off.
PostgreSQL can do so much outside-the-box shit you wouldn't believe, including a lot of stuff people normally go apeshit "scaling out" a Redis or RabbitMQ deployment for where an existing PostgreSQL instance could easily do the job with room to spare and little-to-no performance difference.

Wait until you start screwing around with native JSON document handling and realize PostgreSQL is a better MongoDB than MongoDB. :story:

It would have never occurred to me to run Postgres dockerized... I was under the impression that it should run on bare metal, as God intended.
PostgreSQL is best managed by things like ansible or terraform for actual installation and initial setup, on bare metal as God intended (as you say). Docker introduces a notable, measurable and inescapable network bottleneck. On Kubernetes, it's even worse (upwards of 30% on identical hardware depending on which network snap-in and service mesh you use, and to my knowledge they still haven't fixed it, and likely can't, since mega-hyper-high-speed networking is the purview of dedicated network/switching chips, not software-defined NICs). I've seen bare-metal single-node PostgreSQL instances outperform "clustered" PostgreSQL on Kubernetes just from the overhead being so overwhelming. Maybe eBPF can eventually catch up, but it's not there yet.

If you must Dockerize it, the advise at the start of that message is accurate -- roll your own image instead of relying on the default. It's surprisingly easy to do, and anyone using Docker in production must (IMHO) be well-versed in Dockerfile and docker-compose.yml syntax, be very familiar with their intricacies and be experienced in creating custom images, so this shouldn't be a "big ask" to begin with anyway.

Docker is a very useful tool, but it's spawned a fresh generation of lazy+incompetent sysadmins (a good sysadmin is lazy but competent -- knows his shit well enough to automate it so he never has to do the same thing twice). "Oh just throw it away and spawn a new one lol" is indeed a useful approach ... sometimes ... until you have an ailing system in production that need to diagnose while it's sick. Then you're stuck shelling into a deliberately crippled, tool-less environment often with little more than /lib, /usr/lib, plain /bin/sh and some application binaries, to diagnose problems.

Any PostgreSQL upgrade should involve exactly one moving part at a time, so the updated container image should be identical in all ways to the old one except for the PostgreSQL binary. If something as basic as glibc gets updated as a dependency during that process, you might as well get pg_dump started sooner rather than later because you'll eventually conclude a dump+restore is the best way forward anyway. You're often honestly better off standing up a fresh instance of the new version and replicating into it (as a slave, with the older version the master) and hot-swapping it into live status once the data's migrated; that's about the only way to avoid downtime during major version upgrades (especially on garbage like AWS RDS Aurora; and they're all too happy to charge you double for that extra capacity during the replication stage).

It is not well-suited to the Docker philosophy of just "replace the container to upgrade or if something breaks." The on-disk format is guaranteed never to change over minor version updates, but the trade-off is it's also almost always going to change across major releases for various reasons (reliability, durability and performance, in that order), which makes major version upgrades non-trivial and that goes against "the Docker way." A good sysadmin or devops engineer is suitable for PostgreSQL, but a dedicated old-school DBA is better.[/url]

ETA:
Holy fucking shit. The most recent commit removes a secret token that was getting baked into the CI/CD pipeline every time publicly. Have you just discovered the ultimate jeet repo? "Vibe coded," leaking secrets in CI/CD, NSA-adjacent... the mind boggles:

Jeet coder said:
ci: drop the access token from the libghidra checkout

libghidra is a public repository, so the checkout needs no credential. The explicit token resolved to an empty value and failed every job before configure ran.
 
Last edited:
PostgreSQL can do so much outside-the-box shit you wouldn't believe, including a lot of stuff people normally go apeshit "scaling out" a Redis or RabbitMQ deployment for where an existing PostgreSQL instance could easily do the job with room to spare and little-to-no performance difference.

Inside you are two wolves. One says "why use five tools when this one tool can do all five tasks?" And the other says "tools should do one thing and do it well."

I don't know. Maybe for certain projects the Postgres-for-everything approach makes sense, but I think for most people, if you need Redis for storing cached values, you're probably going to be better served by actual Redis rather than Posgres tweaked and mangled into behaving like Redis. But perhaps straddling the fence and not being dogmatic about either approach is the most wise.
 
Inside you are two wolves. One says "why use five tools when this one tool can do all five tasks?" And the other says "tools should do one thing and do it well."
Use one tool to coordinate the other tools. (usually bash or something like make)
 
I don't know. Maybe for certain projects the Postgres-for-everything approach makes sense, but I think for most people, if you need Redis for storing cached values, you're probably going to be better served by actual Redis rather than Posgres tweaked and mangled into behaving like Redis. But perhaps straddling the fence and not being dogmatic about either approach is the most wise.
If you've got a big enough data set for the distinction to matter, keeping them separate makes sense. Most Redis deployments I've ever seen are over-provisioned, over-engineered front end caches. There are precious few use cases where Redis is substantially faster or better-suited than PostgreSQL, and a mixed tech stack has higher complexity than a simpler one. The correct approach is "PostgreSQL for many things, especially when said 'thing' is 'stuffing data in a database engine for some reason'."

A meaningless anecdotal example I found illustrates the point nicely. Most people aren't using Redis for what it's good at, and those who are tend not to be doing it right (though that's a different argument). The author just built his bog-standard cutesy web app with Redis in the stack because "it's just what you're told to do." Turns out he didn't need it, and things got faster when he got rid of it (not a slam against Redis' performance, which is phenomenal when applied as it's intended to be; more to point out he was "using it wrong" and the solution to his performance woes was "stop doing that," which is more often than not a very good answer to a lot of performance problems). The irony is he was also using PostgreSQL "wrong" by not actually using all its stable features; if he'd known about them up front, he'd have saved himself a lot of headache, development time and hosting costs.

And while I'll grant Redis can still have a valid place in a tech stack where it's actually warranted, I'll happily argue you do not need a separate "document store" like MongoDB or its competitors that make "slinging JSON" their bread-and-butter, as PostgreSQL has first-class, enabled-by-default functionality to do so as well. You get all of SQL's standard syntax and functionality and indexing (via GIN indexes), caching, full-text search and the usual bells and whistles PostgreSQL provides for regular tabular data, along with the usual high performance and reliability. And yes, you can index by deeply-nested keys in complex JSON structures just like MongoDB can. These features are mature, well tested and production-ready. You can add FerretDB to proxy requests to PostgreSQL if you still have clients that must speak MongoDB over the wire. Version 2.0 reached GA in 2025 and is also considered production-ready.

For an extra "fuck you" against MongoDB, nobody should be using it anymore for anything anyway since they went commercial+closed-source. If you must have a MongoDB-alike with wire protocol compatibility, you want DocumentDB now (inb4 "lol it came from Microsoft" ... it's owned by Linux Foundation now and both Amazon and Google also have their tendrils in it and all three big boys are using it for their fancy hosted cloud JSON document stores now, so jeetification/enshitification aside, it works). If you don't need the wire protocol but must maintain a separate "document store" for some reason, go with CouchDB (or ArangoDB if you're allergic to Apache Foundation projects or have byzantine relationship requirements in your data).
 
the host->application model is actually pretty good.

web browser -> websites
bash -> commands

now you're supposed to have os -> gui as an intermediate layer above the shell but below the web browser but all oses are shit, so now all applications are web browsers
 
Sneed Rant:

Version and provision your pipelines themselves too, including testing the pipelines. This really should be done more. You should be able to spin that shit up fast.
Green/Blue or Canary and a load balancer (when you can) or something, goddamn son. We know this.
HIRE A DBA. The number of people who don't know to slap indices on anything never mind what a clustered one is too damn high.
Schema changes should be more than just your dated/numbered sql scripts.

Don't let QA do shit manually. Automate it. If I tell Claude to read QA instructions and script it out in my underwear at 9 pm what is QA doing 9am to 5 pm? The scripts are the best documentation you'll have since product keeping and sharing biz rules is lol

You should be able to go from one region of your host of choice to another, like east vs west, quickly. You might have to. I've seen what happens when you have to "hang on to your butts" instead of run a script for it.

STOP HIRING OFFSHORE PEOPLE, MANAGERS. My entire career is fixing their shit.
 
I'll happily argue you do not need a separate "document store" like MongoDB or its competitors that make "slinging JSON" their bread-and-butter, as PostgreSQL has first-class, enabled-by-default functionality to do so as well.

Maybe it's just me being an old fart but the concept of using JSON as a data store is nonsensical to me. It's a serialization format for sending data along the wire. Once you've received it, you parse it, work with the data inside, then never deal with the JSON again. Now I know Postgres and the others aren't literally storing the data as JSON files internally but still, I don't understand how and why to think of data as being stored that way instead of good ol' table rows.
 
1785993984945.png


Hmmm.

And speaking of Cloudflare, has anyone used Argo tunnel? for something like a cheap-ass VPS.

1785994855499.png
 
Last edited:
Maybe it's just me being an old fart but the concept of using JSON as a data store is nonsensical to me. It's a serialization format for sending data along the wire. Once you've received it, you parse it, work with the data inside, then never deal with the JSON again. Now I know Postgres and the others aren't literally storing the data as JSON files internally but still, I don't understand how and why to think of data as being stored that way instead of good ol' table rows.
Preaching to the choir here, mang. I'm just telling you it's possible and done well with the tool we're discussing.

As for using JSON for document storage in the first place, tell it to the MongoDB/DynamoDB crowd. I've always found the utility of it rather limited too (it feels like an attempt at an end-run around defining a reasonable schema and being too lazy to try to conform and normalize the data you're working with), but lots of people fucking love it for some reason.

"Run our AI gimmick on our infrastructure and rack up billables!" ... standard-procedure AI SaaS from where I'm sitting.

And speaking of Cloudflare, has anyone used Argo tunnel? for something like a cheap-ass VPS.
Not quite what they're offering. The weak link in Cloudflare's main product offering (being a caching anti-bot anti-DDoS web proxy) is they provide free SSL certs and HTTPS by default, but that only covers traffic between the end user and Cloudflare's infrastructure. The unspoken problem (which they're very keen to keep quiet about) is people rarely, if ever, configure encryption between their server and Cloudflare. All the usual howto's and examples and tutorials gloss right over it. So your clients see that fancy little lock in their browser, but the truth of it is their super-special-sekrit traffic goes from e.g. AWS' guts, across the internet over to Cloudflare, over an unencrypted link (usually port 8080 or 8000 if you're dicking around with AWS load balancers and the like for ingress from Cloudflare). Anybody who owns any piece of the infrastructure between AWS and Cloudflare can sniff that traffic and observe its contents. Technically this is often a moot point since Cloudflare's got enough clout now that they've got their tendrils into AWS, Azure and GCP's infrastructure at this point and the whole path is "trustworthy" if not actually encrypted.

Of course, you're fucked if you're just running a handful of real servers in a rack at a data center and trying to front it with Cloudflare if you go with the published defaults, since it leaves that traffic unencrypted between your servers and Cloudflare's stuff.

What they're offering is a bespoke encrypted tunnel for HTTP/HTTPS traffic (only -- it doesn't support anything besides that, on either TCP or UDP). It's not free either. They claim it is, but that's only if you're already paying for Argo routing, which is "free" for up to 1GB of traffic (woo!) and $0.10 per GB afterward (up and down counts). If you're not paying for Argo routing, it's "free" for up to 50 tunnels, with rate limits if they think you're up to something "naughty," and if you want anything fancy added to it (like "zero trust" gimmicks) you're looking at $7/mo per user.

It's useless as a VPN offering; it just secures HTTP(S) traffic between their infrastructure and your workstation and/or your server. Nothing else. So you're not going to be using it for any kind of server that's not speaking HTTP (so no databases, no torrent clients, no RPC, no non-HTTP file transfers, nothing).

Wireguard is best for bog-standard el-cheapo VPN on some server out in the middle of the Pacific ocean or whatever. Set that host up as your "bastion," then connect clients to it so their traffic routes through it. Viola, there's your VPN. For all the world cares, any requests your clients are making are coming from that bastion host. And unless you explicitly configure it to the contrary, all traffic goes over the tunnel when connected, so you're not leaking side traffic.

It even works natively on Android (possibly iOS too, but I don't give a fuck about iPhones or their users since that whole platform is compromised from the ground up anyway ... not that Android's much better, alas), so you can literally route all your phone's traffic through e.g. a piHole instance at home (or on a server somewhere) to nuke ads and tracking crap before it ever reaches your phone, and it can even block Google apps and the like.
 
I appreciated the fact it didn't bundle up updating and fetching together as git does with pull
fetch + merge?

What if, when you wanted a new feature in the software you are using, you could just prompt your agent to add it?
This doesn't work in the cloud Software-as-a-Service model, because you are not running you own copy of the app.
Sandstorm tried to change that 10 years ago, but the world wasn't ready, because not enough people had the skill or patience to actually modify their software. AI has changed that.
webdevs re-inveting open source :story:
 
Maybe it's just me being an old fart but the concept of using JSON as a data store is nonsensical to me. It's a serialization format for sending data along the wire. Once you've received it, you parse it, work with the data inside, then never deal with the JSON again. Now I know Postgres and the others aren't literally storing the data as JSON files internally but still, I don't understand how and why to think of data as being stored that way instead of good ol' table rows.
Because often there is data that doesn't make sense to store relationally and doesn't have any particular set structure. Having a JSON field here makes sense.

I've also used the files as a cache for importers during development to save on API requests. There is no benefit in putting this data in a relational format.
 
Because often there is data that doesn't make sense to store relationally and doesn't have any particular set structure. Having a JSON field here makes sense.

See, that's just it. Why would you not want your data to have a particular set structure? In what practical case is that not a problem that needs to be solved?

I've also used the files as a cache for importers during development to save on API requests. There is no benefit in putting this data in a relational format.

Well, in that case, you might as well save it as a text blob or file on the filesystem, right? You don't need to do queries against the data in it or anything.
 
See, that's just it. Why would you not want your data to have a particular set structure? In what practical case is that not a problem that needs to be solved?
Because often I have no control over the structure, and it could change outside of my control.

Therefore, the only way to deal with it is to accept it as is and then do some wankery (usually reflection) later on.
Well, in that case, you might as well save it as a text blob or file on the filesystem, right? You don't need to do queries against the data in it or anything.
What benefit would that bring? The data is already in JSON; I would convert it to another format and then have to parse it back later from the cache. So I would need two different ways of parsing the same data. This could lead to subtle bugs.
 
Therefore, the only way to deal with it is to accept it as is and then do some wankery (usually reflection) later on.

If you are trying to store data as JSON you need to do some reflection. That we agree on.

What benefit would that bring? The data is already in JSON; I would convert it to another format and then have to parse it back later from the cache. So I would need two different ways of parsing the same data. This could lead to subtle bugs.

Maybe I'm misunderstanding what you said. I thought you were caching JSON responses from servers to reduce on repeated API calls. In that case I wouldn't see why you'd need to manipulate the data.
 
If you are trying to store data as JSON you need to do some reflection. That we agree on.
😒

There are plenty of decent cases where there is literally no point in trying to transform the data into another structure. As previously stated, I often have to deal with data from third parties that can change its shape. Often if I marshal it, I am just marshalling it to something like (in Go):

Code:
type SomeResponse map[string]interface

If I am using a decent SQL database, I have a JSON type (I know it's probably just ntext or whatever underneath) already there.

There are other cases where you just get arbitrary data and you may not get all the fields documented beforehand, and you only find this out by running it in production for a long time.

e.g., I would get live event data from soccer matches. The rest of the database (no-live data) is stored relationally.

Event detail, though, can and often is arbitrary because it just comes from someone on the ground entering it into a computer as the event is taking place.

In Go the structs would look something like this:

Code:
type EventDetail struct {
 // LOADS OF POSSIBLE ARBITRARY FIELDS THAT AREN'T DOCUMENTED
}

type Event struct {
 EventID int
 MatchID int
 Time time.Time
 Detail EventDetail
}

The EventDetail can have a bunch of arbitrary fields or new ones that I won't know beforehand. So while my data is structured, I don't know the full structure.

Decent databases, e.g., MSSQL and Postgres, can deal with JSON properly and can even query it via SQL syntax, and some ORM libraries can deal with it decently as well.
To preserve all data, I would need to implement the EAV database structure, which is often considered an anti-pattern.

The data attached to this is kind of only sensible to keep in some K/V value format, and most SQL databases have a JSON format, and it comes from their server as JSON; I am going to be sending it to the client as JSON directly while the match is live. There is literally no point in transforming it; it makes the database queries more complex, as it often can't be expressed sensibly relationally.

I, however, can store it as JSON (as it basically is) and tell my ORM that the type is stored as JSON, and it can deal with it, e.g., update the code later to marshal known fields into my EventDetail type.

This way I capture all the data, and if there are any fields present that aren't in my model, I can adapt it later without any event data loss.
Maybe I'm misunderstanding what you said. I thought you were caching JSON responses from servers to reduce on repeated API calls. In that case I wouldn't see why you'd need to manipulate the data.
I misunderstood what you were saying.
 
Last edited:
Back
Top Bottom