# emacs-piper

**Repository Path**: feng-qi/emacs-piper

## Basic Information

- **Project Name**: emacs-piper
- **Description**: Clone of https://gitlab.com/howardabrams/emacs-piper
- **Primary Language**: Unknown
- **License**: GPL-3.0
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2021-02-14
- **Last Updated**: 2021-02-14

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

#+TITLE:  Piper Project for Emacs
#+AUTHOR: Howard Abrams
#+DATE:   2019-09-03 September

*tl;dr* ...
Wrapper around existing Emacs functionality that creates an interactive user interface to dealing with unstructured textual data similar with how we transform data through pipes in the shell.

Whew.

  Let me begin by stating that what follows is an idea...not my go-to approach.
  However, I daresay, this idea has some legs.  So, I'm making it [[https://gitlab.com/howardabrams/emacs-piper][publicly accessible]] and discussing it at EmacsConf 2019.  I'm hoping to generate some discussion.  Let me attempt to explain the details...

* The Itch

  "Hey Howard, can you restart all the OpenStack services on that controller?"
  Uh, sure.  I'm going to assume that all OpenStack services begin with the word, =openstack=, so I immediately typed:

  #+begin_src shell
    for S in $(systemctl --all | grep openstack | sed 's/\.service.*//' | cut -c3-)
    do
        systemctl restart $S
    done
  #+end_src

  Yes, I know.  Hacky, but we do this stuff all the time in the shell, right?  Now, if I banged this out the first time, I wouldn't complain.  But that /sub-shell sequence/ within the =$(...)= took a few iterations to get right.

  You know how this goes.   Start with an initial command, like =systemctl=, and gaze at the output,  adding one pipe after another, perfecting the last command before moving to the next piped command.  With the correct output, we're ready to move to the =for= loop.

  Since we don't have a /shell command sequence debugger/ (I don't even know what that would look like), I want the text that flows from /standard out/ to /standard in/ to be more visible.
  A more /iterative approach/ without the constant calling of the initial executable (in my example,  =systemctl=).  The typical approach we use is to simply write the output into a file, and work from that.

  But I'm in Emacs.

  Seems a more Emacsy way would place the output of the =systemctl= command into an Emacs buffer, kick off a few functions to modify the data and then ship the entire buffer to another instance of =systemctl=, but this time with a =restart= option, and pass it through =xargs=.

* The Scratch (or Usage)

  Piper works upon a simple =input -> operation -> output= model that
  intertwines shell commands with Emacs buffers.

** Gathering Data

   This project has five entry points that run a shell command placing the output in a buffer for editing:

  - =piper= :: Runs a command in local directory and start my user interface on results (see below).
  - =piper-other= :: Same as =piper= except it prompts for a directory.
  - =piper-remote= :: Same as above, but it also prompts for a remote host, and executes the command there.
  - =piper-user-interface= :: Summons the piper interface for the current buffer.
  - =piper= for eshell :: with a shell command, but the results overlay the eshell buffer and starts the user interface.

  After the buffer contains the output, we start a /user interface/ that is simply a Hydra to present a collection of useful functions grouped into the following categories:

  - Movement (the basics of Evil and Emacs keybindings)
  - Transforming the data and interacting by executing shell commands (see next section)
  - Processing the data as a series of lines (since both =keep-lines= and =flush-lines= do not have default keybindings).
  - Standard transformations typical of the shell command output, e.g.  =cut=

** Transforming Data

   I'll admit, I think about /transforming/ data while in a shell, and /editing/ data when I'm in my editor.  When dealing with a buffer of data from =systemcl=, will I think of transforming with =grep= or /editing/ with =keep-lines=?  I'm not sure.  So this =piper= project does both.

   Here is an example of the shell snippet shown above with data transformation by calling the commands, =grep=, =sed=, etc:

   [[file:piper-demonstration-1.gif]]

   In the animation above, typing a ~|~ character, sends the contents of the buffer to a shell command, replacing the buffer with the results.  This allows you to essentially type a typical shell command line full of pipes, but /watch the data transformations/ as they happen.

   Here is the same example, but this time /edited/ with =keep-lines=, =vr/replace=, etc:

   [[file:piper-demonstration-2.gif]]

   The end result is the same.

** Sending Data

   We need to make both the pipe transformation process (described above) and the consumption by executables /seamless/.  For this last part, I can see the following solutions cover 80% of my use cases:

   - Copy the contents of the buffer into the clipboard and close the window.  See both the ~y~ and ~Y~ commands.

   - Send buffer contents as standard input to another executable or shell command/script, for instance:
     #+begin_src shell
       $(buffer-contents) | xyz
     #+end_src
     See the ~!~ and ~|~ commands.

   - Send buffer contents as a series of argument parameters to an executable, for instance:
     #+begin_src shell
       xyz $(buffer-contents)
     #+end_src
     Or a more shell-programmatic way:
     #+begin_src shell
       $(buffer-contents) | xargs xyz
     #+end_src
     See the ~x~ command.

   - Calling an executable on each line, similar to what we do with loops, for instance:
     #+begin_src shell
       for LINE in $(buffer-contents)
       do
         xyz $LINE
       done
     #+end_src
     See the ~f~ command.  Note, at the moment, you can only pass it to a single executable.  We should allow a full shell command and even something similar to eshell's filename modification feature.

   While many commands expect a file name as the final parameter, many do not.  So, by default, the ~f~ command appends each line to a repeated shell command, and if you need to embed the buffer line in the middle of the shell command, we emulate the =find= command, where the =-exec= parameter replaces a sequence ={}= with the filename.

  For instance, if I had a buffer with the contents:

  #+begin_example
    one fish
    two fish
    red fish
    blue fish
  #+end_example

  Hit the ~f~ and typed:

  #+begin_src shell
    echo start {} and end
  #+end_src

  I would have a new buffer containing:

  #+begin_example
    start one fish and end
    start two fish and end
    start red fish and end
    start blue fish and end
  #+end_example

  Also notice the difference between the /red/ and /blue/ commands.  The red commands call a function, but return to this hydra, allowing me to type ~K~ a couple of times to run the =keep-lines= functions for multiple transformations, however, the commands mentioned here will end the UI dropping you into a regular Emacs buffer.


* Piper, the Script

  One of the things that gives the shell its power, is the ability to extend it with scripts, and shell scripts /can/ often be short and readable... however, without any data structures (and other limitations), getting the shell to do anything more complex than a loop, can create an unreadable mess.  How about we take the best aspects of the shell and bring it into Emacs Lisp?

  For example:

  #+begin_src emacs-lisp
      (piper-script
       ($ "ls" "-1" "~/other/*")
       (| "grep Temp")
       (| "tr [a-z] [A-Z]"))
  #+end_src

  Returns a string of the output of the last command.  The process acts the same as the =piper= features we've shown above, as each command reads and writes into a buffer.  No reason why we can't mix and match between shell commands and Lisp functions:

   #+begin_src emacs-lisp
     (piper-script
      (sudo
       (cat "/proc/acpi/wakeup")
       (grep "enabled")
       (replace-regexp " .*" "")
       (for (device (read-all-lines))
         (write-into device "/proc/acpi/wakeup"))))
   #+end_src

   The /trick/ is that =piper-script= is a macro that converts any strings:
   - Expands both =$VAR1= and =${VAR2}= to match either environment variables or Emacs variables
   - Expands file wildcard references
   - Converts initial =~/= into the files from the Home directory
   - Makes all files absolute...which seems to make it easier when running over Tramp

   The other trick is making short "functions", like =cat=, as an alias for longer functions, =insert-file-contents=, in this case.  Along with being shorter, they look more /shell-like/.  Keep in mind, that currently, you can't have any variables that match any of these functions (lots of improvements to make).

   Currently supported:

   - =sh= :: Call a shell script, aliases include =shell=, =$=, and =|= (see notes below)
   - =sudo= :: Runs the commands given to it with root privileges by changing the =default-directory= to include a Tramp =/sudo:= reference.
   - =echo= :: Replaces the buffer contents with the strings given to it
   - =setenv= :: Sets an environment variable
   - =source= :: Read a file and sets any key=value pairs as environment variables
   - =ifsh= :: Like =if=, but the condition comes from the command's exit code (this needs work)
   - =for= :: Like =dolist= where it iterates over some forms setting a variable to each element from a list, see =read-all-lines= as a replacement for shell's =for= loop.
   - =grep= :: An alias for =keep-lines=
   - =grep-v= :: An alias for =flush-lines=
   - =replace= :: Replaces all occurrences matching a regular expression, alias with =replace-regexp=
   - =sort= :: An alias for =sort-lines=
   - =sort-r= :: Sorts all lines in the buffer in reverse order
   - =uniq= :: Acts like the =uniq= command line executable
   - =cat= :: An alias for =insert-file-contents=
   - =write-into= :: Writes a script into a file
   - =to-clipboard= :: Copies the current, temporary buffer into the kill-ring
   - =read-all-lines= :: Creates a list of all lines in a current buffer

   The shell behaves a little differently.  Currently, if you give it a single string, it behaves like the shell, and breaks the text into parameters based on spaces, however, if you give it a series of strings, it doesn't bother, and honors the spaces (in other words, no need to quote them).  For instance:

   #+begin_src emacs-lisp
     (piper-script
      ($ "ls ~/Google Drive File Stream"))
   #+end_src

   Doesn't work as it passes four parameters to the "ls" command.  On the other hand, the following works as expected:

   #+begin_src emacs-lisp
     (piper-script
      ($ "ls" "~/Google Drive File Stream"))
   #+end_src

   Yeah, this looks like what you'd expect.

* Installation

  I don't feel comfortable releasing this on Elpa just yet.  So either clone this repository and add the following:

  #+begin_src emacs-lisp
    (use-package piper
        :load-path "~/Other/emacs-piper/"
        :bind ("C-c C-|" . piper))
  #+end_src

  For those using the [[https://github.com/raxod502/straight.el][straight]] package manger, you can install *piper* using the following recipe.

  #+begin_src emacs-lisp
    (straight-use-package
     '(emacs-piper :type git :host gitlab :repo "howardabrams/emacs-piper"))
  #+end_src

  Or if you are using Spacemacs and want a leader key:

  #+begin_src emacs-lisp
    (use-package piper
        :load-path "~/Other/emacs-piper/"
        :config
        (spacemacs/declare-prefix "o |" "piper")
        (spacemacs/set-leader-keys
         "|"     '("piper-ui"        . piper-user-interface)
         "o | |" '("piper-locally"   . piper)
         "o | d" '("other-directory" . piper-other)
         "o | r" '("piper-remotely"  . piper-remote)))
  #+end_src

* Development

  As I described at EmacsConf 2019, this is currently a project /for tinkerers only/.
  Nothing in this project is stable and everything is still just an idea, so if interested, let's collaborate.  You'll need to begin by reading the source code:

  - =piper.el= :: The interactive piper bits
  - =piper-script.el= :: The bits about the scripting language DSL
  - =piper-operations.el= :: Any code shareable between the two
  - =source-environment.el= :: Code for reading and parsing shell files looking for key=value pairs, so that I could =source ~/.resource= in my shell scripts

  The unit tests are in the =test= directory, and use =cask= to run them:

  #+begin_src shell
    cask exec ert-runner
  #+end_src

  Hope this looks fun!