Optimizations - pipecut benefits from an Abstracted view of Unix commands

Every blade in pipecut is internally seen as an element in an abstract syntax tree.

AST representation of Unix commands

Internal vs BlackBox Blades

Pipecut has internal implementations of some common Unix utilities. This may be something you don't want to use - and later on it will probably be possible to disable these entirely. They were originally written for one reason, and continue to exist for two more reasons.

Internal representations of commands were useful in the initial development of pipecut, when I wrote the AST functions, and tested the UI before I implemented support for Black-box commands. Black-box commands are neccesary, since without them, pipecut would have a very limited command vocabulary, and would only be usable for the most basic command lines. Before Black-box support was implemented, I didn't have an intuitive sense for how much slower it might be to be spawning external commands to generate the blade caches, and I wanted to test the UI with commands that were very fast.

For the purpose of generating its own regression tests, pipecut needs a baseline to compare against the system utilities that are native on the platform it is running on. So it is useful to have internal versions of commands to support that.

Further down the roadmap, when pipecut can generate code in other languages, for that code to consist of more than just calling pipe(2) and fork(2) to run Unix commands, there need to be some 'atoms' of data transformation which can be implemented natively in the target languages. This is comparable to a very high-level 'bytecode'. Regression tests will be even more important for making sure the code-generators are working correctly.

One of the objectives of pipecut is to use the AST knowledge of the command pipeline to execute the requested commands, but after optimizing them. This can shorten them in terms of runtime or length.

To shorten the processing time for running a toolset, we can use the meta-information in the AST to short-cut certain operations.

Pipecut was first displayed in a presentation Nov 1, 2014 at MeetBSD California. Pipecut was created by David W. Maxwell