Programming thread

[Yotsubaaa]

If programs were functions, they would compose just like any other function with even less ceremony:

bar x | foo

is just

foo(bar(x))

I appreciate that I'm in a minority here, and lots of people think that UNIX pipes are cool. To be fair, it is a good idea in a highly networked world to make sure you've made streaming data a priority. But aside from streams, the fact that anyone thinks UNIX pipelines are pleasant is proof of how brain-damaged UNIX has made us, and apparently this brain-damage is so bad that it's infected Powershell.

I mean, what the hell is the thought process?

"We have small units of functionality in UNIX called programs. These programs have an input and output."

Okay. So far so good. You've got functions.

"These programs input and output text."

Okay, hang on.

"To send the output of one program into another program, you ask the kernel to create a special data structure called a pipe with an input and an output end. You tell the first program that its output is the input of this pipe, and the second program that its input is the output of this pipe."

Err...what the fuck? If you want to send the output of a function f to a function g, why can't you just do g(f(x)).

I dunno. I sort of agree with you, but then again I just finished an Arch Linux reinstallation last night (went okay!) and one of the system admin pages of the ArchWiki had something interesting to say about the UNIX "Everything is a File!" mantra that's been fresh on my mind. (And yes, I'm that dork who reads the ArchWiki cover to cover while doing an install...)

For example, it isn't just programs and data. The file metaphor also extends directly to hardware peripherals:

... One of the most important of these is probably the mantra: "everything is a file," widely regarded as one of the defining points of UNIX. This key design principle consists of providing a unified paradigm for accessing a wide range of input/output resources: documents, directories, hard-drives, CD-ROMs, modems, keyboards, printers, monitors, terminals and even some inter-process and network communications. The trick is to provide a common abstraction for all of these resources, each of which the UNIX fathers called a "file." Since every "file" is exposed through the same API, you can use the same set of basic commands to read/write to a disk, keyboard, document or network device.

A simple tool, such as cat, designed to read one or more files and output the contents to standard output, can be used to read from I/O devices through special device files, typically found under the /dev directory. On many systems, audio recording and playback can be done simply with the commands, "cat /dev/audio > myfile" and "cat myfile > /dev/audio," respectively.

So in that regard, it seems pretty elegant: as long as your 'thing' that you want to interact with—virtual or physical—can be crudely bent into the same file metaphor (can you 'write' to it, whatever that means? And can you 'read' bytes from it, whatever that means in your context?), then all of a sudden all of the UNIX tools can work with it with 'no dramas', and you can chain them together with pipes and have fun and all that. Well, in ideal, anyway.

Of course, I'm not fangirling for UNIX or anything! I still have the same reservations that I imagine you do: the function (or object) abstraction seems like it would actually be simpler and more flexible these days. But I guess for its time, back when all of the hardware peripherals were simpler too, the file metaphor was a neat way to get everything conforming under a simple abstraction, without having to worry about adapters for incompatible interfaces or anything like that (much).

 

[cecograph]

So in that regard, it seems pretty elegant: as long as your 'thing' that you want to interact with—virtual or physical—can be crudely bent into the same file metaphor (can you 'write' to it, whatever that means? And can you 'read' bytes from it, whatever that means in your context?), then all of a sudden all of the UNIX tools can work with it with 'no dramas', and you can chain them together with pipes and have fun and all that. Well, in ideal, anyway.

Yes, UNIX observed that any data can be turned into a byte stream and then built a load of stream processing tools. But this seems about as silly as observing that any data can be turned into a huge number and then building a calculator.

We want byte streams, of course. In fact, we want far more than just byte streams. For any kind of data, I want to be able to talk about a stream of that kind of data. And I want to be able to create streams from sources and feed streams into sinks, as in the UNIX audio example. Any operating system should expose a language for doing this elegantly. But it should have many more data abstractions than that.

Of course, I'm not fangirling for UNIX or anything! I still have the same reservations that I imagine you do: the function (or object) abstraction seems like it would actually be simpler and more flexible these days. But I guess for its time, back when all of the hardware peripherals were simpler too, the file metaphor was a neat way to get everything conforming under a simple abstraction, without having to worry about adapters for incompatible interfaces or anything like that (much).

Maybe that's it. There's a famous series of essays suggesting that UNIX succeeded because it never tried to be too smart, and as a result, it was much easier to port to new hardware than its supposedly superior competitors. And once it was easy to port, it could spread quickly and take over, leaving us where we are today.

If you want some more reading and some giggles, you might check out The UNIX hater's handbook if you haven't already. It's an anthology of early 90s mailing list posts, hating on UNIX. One big takeaway I get is that the hater's were familiar with alternative operating systems that they considered superior, and which are now presumably forgotten.

Congrats on the Arch install. I was a Gentoo user for ages, and still love my Linux minimalism.

 

[Splendid]

@Yotsubaaa the problem with device files is that they predate pretty much every peripheral that we have nowadays. Mice didn't exist until later, and even our modern printers don't can't do well with text (notice how the unix print command is lpr, or Line PRinter. You won't really see those around anymore.)
In my dream world, unix fucking died and we got some sort of kickass FOSS clone of VMS or hell even windows NT instead (although in the latter case we might need another OS for certain high uptime servers, maybe an AS400 clone?)
Basically, I'm just grumpy that there are 2 and a half (MacOS being the half) operating systems in widespread use outside of embedded applications and mobile devices, and that mobile devices tend to just be a sophisticated way of running web browsers since users rightly do not trust most native apps with a lot of access to their phones. I think we should have more OS's for specific purposes.

 

[Shoggoth]

If programs were functions, they would compose just like any other function with even less ceremony:

bar x | foo

is just

foo(bar(x))

I appreciate that I'm in a minority here, and lots of people think that UNIX pipes are cool. To be fair, it is a good idea in a highly networked world to make sure you've made streaming data a priority. But aside from streams, the fact that anyone thinks UNIX pipelines are pleasant is proof of how brain-damaged UNIX has made us, and apparently this brain-damage is so bad that it's infected Powershell.

I mean, what the hell is the thought process?

"We have small units of functionality in UNIX called programs. These programs have an input and output."

Okay. So far so good. You've got functions.

"These programs input and output text."

Okay, hang on.

"To send the output of one program into another program, you ask the kernel to create a special data structure called a pipe with an input and an output end. You tell the first program that its output is the input of this pipe, and the second program that its input is the output of this pipe."

Err...what the fuck? If you want to send the output of a function f to a function g, why can't you just do g(f(x)).

Hold up, I didn't say serializing data between processes was a good idea, and parsing it is even worse. The idea is that queues are excellent for separating processes, and letting them communicate by channels (hmmm... CSP) is genius. UNIX had to go and be weird about it, but the very idea is good. The strength of f x | g over (g(f(x)) is exactly for stream processing and even driven designs. f and g can run in different threads and be independent of one another. If f is twice as fast as g you can dedicate two times the number of threads to achieve optimal output.
Or maybe my head is just full because I read the CSP article over the weekend and have been getting into Clojure's core.async deeply in the past two months.
TLDR - the cool part about unix pipes is the pipes, not the unix.

 

[Least Concern]

the UNIX policy is that programs should not share memory directly, but instead waste CPU cycles spitting out their data into a bespoke text format and then expect every other program to waste cycles with bespoke, often catastrophically buggy, parsers to accept text in these formats.

You're so right. Having all these tools exporting bespoke object formats instead would have been so much more efficient and less prone to bugs.

If programs were functions, they would compose just like any other function with even less ceremony:

bar x | foo

is just

foo(bar(x))

I appreciate that I'm in a minority here, and lots of people think that UNIX pipes are cool. To be fair, it is a good idea in a highly networked world to make sure you've made streaming data a priority. But aside from streams, the fact that anyone thinks UNIX pipelines are pleasant is proof of how brain-damaged UNIX has made us, and apparently this brain-damage is so bad that it's infected Powershell.

I mean, what the hell is the thought process?

I can't be sure, but I think part of the process is that, as crazy as it sounds by today's standards, UNIX was, in its time, designed to be a user-friendly system for non-programmers. And non-programmers can understand reading from left to right and the concept of a pipe (don't think of that in computer terms; think of it as an actual metal tube) to connect the output of something to the input of something else, and for these people who haven't taken any courses in programming and maybe not even courses in calculus, foo x | bar is probably more legible than bar(foo(x)) which is "backwards" and has more "symbols."

Also, this whole C thing was kind of new. The world hadn't yet settled on something like foo(x) as being immediately recognizable as a function call with a passed parameter, or even the concept of "functions" as we know them today.

In my dream world, unix fucking died and we got some sort of kickass FOSS clone of VMS or hell even windows NT instead (although in the latter case we might need another OS for certain high uptime servers, maybe an AIX clone?)

There's an OS called Plan 9 which was intended to be a sort of sequel to UNIX and improve on its ideas, though it never really caught on and currently lives on at the "research OS" level (meaning people hack on it to learn about OS development but nobody really uses it full-time due to its limited software support and functionality). It's an example of "new and improved" not being strong enough to conquer "established and good enough." As for a WinNT clone, there's ReactOS, a FOSS OS which implements the Win32 API so you can run Windows software without Windows. It sometimes works.

 

[cecograph]

Hold up, I didn't say serializing data between processes was a good idea, and parsing it is even worse. The idea is that queues are excellent for separating processes, and letting them communicate by channels (hmmm... CSP) is genius. UNIX had to go and be weird about it, but the very idea is good. The strength of f x | g over (g(f(x)) is exactly for stream processing and even driven designs. f and g can run in different threads and be independent of one another. If f is twice as fast as g you can dedicate two times the number of threads to achieve optimal output.
Or maybe my head is just full because I read the CSP article over the weekend and have been getting into Clojure's core.async deeply in the past two months.
TLDR - the cool part about unix pipes is the pipes, not the unix.

I like streams and async. But I don't credit UNIX for them. Should I?

You're so right. Having all these tools exporting bespoke object formats instead would have been so much more efficient and less prone to bugs.

There's no comparison. An object isn't a format. It's just the data as it is in memory. In terms of efficiency, working with an object is just peeking with offsets or pointer chasing, which is infinitely faster than parsing a bytestream. And if you're in something like .NET, you shouldn't even be allowed to run your command over the data unless your code typechecks, which is infinitely safer. Even in untyped languages like Lisp, you're going to get fewer bugs asking for the field named "foo" in the second element of a vector than you are figuring out what the fuck to do when you've just been given a load of text and an environment variable shrugging "maybe this is a field delimiter?"

And non-programmers can understand reading from left to right and the concept of a pipe

Personally, I give more credit to non-programmers, who should have a basic high-school maths education. If you can use excel, you can understand functions.

 

[Shoggoth]

I like streams and async. But I don't credit UNIX for them. Should I?

I don't think so, but I'm not familiar enough with the history of computing to know who gets the credit.

If you can use excel, you can understand functions.

How many people can use excel? I think the numbers are depressingly low.

Maybe you'll enjoy this project

dundalek/closh

Bash-like shell based on Clojure. Contribute to dundalek/closh development by creating an account on GitHub.

github.com

 

[Least Concern]

How many people can use excel? I think the numbers are depressingly low.

Are you kidding? Excel makes the business world go round. If you really wanted to watch the world burn, create a virus which does nothing else but deletes Excel from peoples' computers and makes it unable to be reinstalled. The world economy would collapse overnight. The argument that the rise to prominence of Excel has introduced more people to the concepts of functions is well made, but Unix also predates Excel by a couple decades.

 

[Shoggoth]

Are you kidding? Excel makes the business world go round. If you really wanted to watch the world burn, create a virus which does nothing else but deletes Excel from peoples' computers and makes it unable to be reinstalled. The world economy would collapse overnight. The argument that the rise to prominence of Excel has introduced more people to the concepts of functions is well made, but Unix also predates Excel by a couple decades.

It does, but most people are not in the business world. Most of the people using excel, too, don't really know how to use it besides the most basic operations, and every excel file which lives more than a month turns into a botched abortion. They can barely operate it.

 

[⠠⠠⠅⠑⠋⠋⠁⠇⠎ ⠠⠠⠊⠎ ⠠⠠⠁ ⠠⠠⠋⠁⠛]

It does, but most people are not in the business world. Most of the people using excel, too, don't really know how to use it besides the most basic operations, and every excel file which lives more than a month turns into a botched abortion. They can barely operate it.

I have found that the least unmaintainable/unfixable Excel sheets are often those that only have one function operating on existing imported/entered data per calculated column. Yes, in theory, you can indent your three level IF(ISNA(SUMIF(.. abortion, but, will you or anyone else ever have to understand it again? If so, just make more columns.

Effectively, Excel pipelines

 

[cecograph]

I don't think so, but I'm not familiar enough with the history of computing to know who gets the credit.

It'd still be cool to see some comparisons, if you've got them, since I'm curious about process algebras, and know shit-all about them. The sort of async code I've written with channels and lightweight threads doesn't feel like it owes much to the sort of things I do in BASH. But this is getting outside my familiarity, and I'm very much prepared that the async libraries I've used suck compared to a process algebra.

On the general thing that functions read right-to-left, I used to be pretty happy with that when I studied normal maths, and only got pissed off with it when I started learning category theory. That f . g diagrams as --g--> --f--> still trips me up.

I can't find it now, but there was an introductory text by a category theorist who said "fuck it", and flipped the orders of application and composition.

In Ocaml, I like to use the pipe operator (|>) to flip application, especially in stream code:

0 -- 10 |> scan (+) |> iter (Printf.printf "Total so far: %d\n")

Maybe you'll enjoy this project

dundalek/closh

Bash-like shell based on Clojure. Contribute to dundalek/closh development by creating an account on GitHub.

github.com

There's also the scheme shell from back in the 90s. It seems like a right of passage for every language community to, at some point, decide that they can reinvent shells. A big Haskell coder, who is also responsible for its best stream library, came up with one called Turtle, but I honestly find it horrible to work with. My smug Lisp weenie conclusion is that the solutions suck because the problem shouldn't have existed in the first place.

 

[Shoggoth]

It'd still be cool to see some comparisons, if you've got them, since I'm curious about process algebras, and know shit-all about them. The sort of async code I've written with channels and lightweight threads doesn't feel like it owes much to the sort of things I do in BASH. But this is getting outside my familiarity, and I'm very much prepared that the async libraries I've used suck compared to a process algebra.

On the general thing that functions read right-to-left, I used to be pretty happy with that when I studied normal maths, and only got pissed off with it when I started learning category theory. That f . g diagrams as --g--> --f--> still trips me up.

I can't find it now, but there was an introductory text by a category theorist who said "fuck it", and flipped the orders of application and composition.

In Ocaml, I like to use the pipe operator (|>) to flip application, especially in stream code:

0 -- 10 |> scan (+) |> iter (Printf.printf "Total so far: %d\n")

There's also the scheme shell from back in the 90s. It seems like a right of passage for every language community to, at some point, decide that they can reinvent shells. A big Haskell coder, who is also responsible for its best stream library, came up with one called Turtle, but I honestly find it horrible to work with. My smug Lisp weenie conclusion is that the solutions suck because the problem shouldn't have existed in the first place.

Here you go

π-calculus - Wikipedia

en.wikipedia.org

Communicating sequential processes - Wikipedia

en.wikipedia.org

 

[Splendid]

You're so right. Having all these tools exporting bespoke object formats instead would have been so much more efficient and less prone to bugs.

With an object, the only option almost all of the time is you pass a valid object or you crash. No interpretation needed.

There's an OS called Plan 9 which was intended to be a sort of sequel to UNIX and improve on its ideas,

Plan 9 isn't nearly radical enough IIRC. I'm talking things like getting rid of some of the excessive modularity in Nix by specifying a single reference window manager (others can still exist, but there would be one "main" one) which would cut down on duplication of effort.

As for a WinNT clone, there's ReactOS, a FOSS OS which implements the Win32 API so you can run Windows software without Windows. It sometimes works.

ReactOS is a meme.

 

[AssRapistDude1488]

Anyone have book recommendations for learning x86 assembly?

 

[Splendid]

Don't

 

[AnOminous]

Learn 6502 assembly instead. I heard it has a huge future.

 

[cecograph]

Twenty years ago, I recall going online, filling out some form, and then getting four heavy books of the Intel x86 developers manual sent to me free.

They weren't very useful for beginners. But I found early editions of this book to be very good.

 

[Least Concern]

I wonder if it would make sense to first learn assembly on something cheaper and simpler like Atmel AVR processors (what Arduino uses) or even ARM and then transfer that knowledge to Intel chips and instructions.

 

[FuckedUp]

Anyone have book recommendations for learning x86 assembly?

https://cs.brown.edu/courses/cs033/docs/guides/x64_cheatsheet.pdf

Pretty much how I learned it.

 

[eternal dog mongler]

Anyone have book recommendations for learning x86 assembly?

It's 2019 dude