管道命令链，每个输出状态到标准错误

Question

我在 bash 脚本中有一系列管道命令，将标准输出管道到标准输入：

prog1 | prog2 | prog3

他们每个人都输出一些标准错误。他们中的一些输出覆盖前一行，一些没有，有些两者都做：例如输出几行输出，然后在shell中有一个更新的“状态栏”。例如，curl 可以将下载进度输出为状态栏。

输出相当不清楚，因为状态栏可能会在一个进程的输出和另一个进程的输出之间闪烁。

有没有办法让各种输出更清晰，例如

• 明确哪个输出行来自链中的哪个程序？
• 让所有状态栏同时可见，不闪烁？

闪烁示例：

Answer 1

尝试这个：

1. 从每个进程输出中删除回车。有时您可能需要用回车符替换换行符。如果颜色不重要，你可以只是cat -v它。
2. 强制行缓冲。（这实际上只是（也许）管道中的最后一个程序需要，但它可以帮助我调试）。

{ stdbuf -oL prog1 | stdbuf -oL prog2 | stdbuf -oL prog3 | stdbuf -oL tr -d '\r' ;} 2> >(stdbuf -oL tr -d '\r'>&2)

在处理多个程序时，我通常为它们的每个输出添加一个标签/前缀，这样我就知道哪一行来自哪个程序：

stdbuf -oL prog1 2> >(sed 's/\r//g; s/^/prog1: /' >&2) |
stdbuf -oL prog2 2> >(stdbuf -oL tr '\r' '\n' | sed 's/^/prog2: /' >&2) |
stdbuf -oL prog3 2> >(sed 's/\r//g; s/^/prog3: /' >&2) |
stdbuf -oL sed 's/\r//g; s/^/out: /'

对于您确实需要为多个进程共享屏幕（并且您正在交互地运行命令）的任何更复杂的事情，使用screen或tmux或类似的方法通过多个进程共享屏幕或编写您自己的应用程序来处理终端：

tmpd=$(mktemp -d)
mkfifo "$tmpd"/1 "$tmpd"/2
trap 'rm -r "$tmpd"' EXIT
# prog1 = seq 5
# prog2 = grep -v 3
# prog3 = cat
tmux new-session \; \
  send-keys "seq 5 > $tmpd/1" C-m \; \
  split-window -v \; \
  send-keys "grep -v 3 < $tmpd/1 > $tmpd/2" C-m \; \
  split-window -v \; \
  send-keys "cat < $tmpd/2" C-m \; \
  select-layout even-vertical \;

但是，如果您打算以非交互方式运行程序，并且仍希望以非易失性方式保留（大量）日志记录信息，我建议使用专为这种情况设计的系统记录器。从外壳使用logger.

$ runlog() { stdbuf -oL "$@" 2> >(logger -p local3.info -t "$1") | stdbuf -oL tee >(logger -p local3.info -t "$1"); }; 
$ runlog seq 3 | runlog grep -v 3 | runlog cat
1
2
$ sudo journalctl -p info -b0 -tseq
-- Logs begin at Fri 2018-11-02 02:06:41 CET, end at Fri 2020-05-08 14:40:24 CEST. --
maj 08 14:39:41 leonidas seq[255641]: 1
maj 08 14:39:41 leonidas seq[255641]: 2
maj 08 14:39:41 leonidas seq[255641]: 3
$ sudo journalctl -p info -b0 -tgrep
-- Logs begin at Fri 2018-11-02 02:06:41 CET, end at Fri 2020-05-08 14:40:14 CEST. --
maj 08 14:39:41 leonidas grep[255647]: 1
maj 08 14:39:41 leonidas grep[255647]: 2

更高级的版本可以使用fifos 和systemd插入单元，这将允许真正微调每个可执行文件的执行。

Answer 2

Interesting ideas have been given here for this challenging question, but I didn't see any complete solution up to now. I will try to give one. In order to achieve this, I first wrote three scripts corresponding to the pipeline prog1 | prog2 | prog3 the PO was speaking about.

prog1 producing messages separated by \n on the error stream and generating numbers on the standard stream:

#!/bin/bash

cmd=$(basename $0)

seq 8 |
while ((i++ < 10)); do
  read line || break
  echo -e "$cmd: message $i to stderr" >&2 
  echo $line
  sleep 1
done

echo -e "$clearline$cmd: has no more input"  >&2 

prog2 producing messages separated by \r and overwriting its own outputon the error stream and transferring numbers from the standard input stream to the standard output stream:

#!/bin/bash

cmd=$(basename $0)
el=$(tput el)

while ((i++ < 10)); do
  read line || break
  echo -en "$cmd: message $i to stderr${el}\r" >&2 
  echo $line
  sleep 2
done

echo -en "$clearline$cmd: has no more input${el}\r" >&2 

and finally prog3 reading from the standard input stream and writing messages to the error stream in the same way as prog2:

#!/bin/bash

cmd=$(basename $0)
el=$(tput el)

while ((i++ < 10)); do
  read line || break
  echo -en "$cmd: message $i to stderr${el}\r" >&2 
  sleep 3
done

echo -en "$clearline$cmd: has no more input${el}\r"  >&2 

Instead of invoking this three scripts as

prog1 | prog2 | prog3

We will need a script to invoke this three programs redirecting the error stream to three FIFO special files (named pipes), but before launching this command, we will have to create first the three special files and to launch in the background processes to listen to the special files: every time a full line is sent, these process will print it on a special area of the screen that I will call a taskbar.

The three taskbars are in the bottom of the screen: the upper one will contain the messages of prog1 to the error stream, the next one will correspond to prog2 and the last one in the bottom will contain the messages from prog3.

At the end, the FIFO files will have to be removed.

Now the tricky parts:

1. I found no utility reading without buffering a line ending with \r, so I had to change the \r into \n before printing the message lines to the screen;
2. some program in the several programs I was connecting with pipes were buffering their input or output causing the messages not to be print until the end, which is obviously not the intended behaviour; for fixing this, I had to use the command stdbuf with the tr utility;

Putting all together, I implemented next script, which is working as intended:

#!/bin/bash

echo -n "Test with clean output"
echo;echo;echo        # open three blank lines in the bottom of the screen
tput sc               # save the cursor position (bottom of taskbars)
l3=$(tput rc)                       # move cursor at last line of screen
l2=$(tput rc; tput cuu1)            # move cursor at second line from bottom
l1=$(tput rc; tput cuu1; tput cuu1) # move cursor at third line from bottom
el=$(tput el)         # clear to end of line
c3=$(tput setaf 1)    # set color to red
c2=$(tput setaf 2)    # set color to green
c1=$(tput setaf 3)    # set color to yellow
r0=$(tput sgr0)       # reset color

mkfifo error{1..3}    # create named pipes error1, error2 and error3

(cat error1 | stdbuf -o0 tr '\r' '\n' | 
  while read line1; do echo -en "$l1$c1$line1$el$r0"; done &)
(cat error2 | stdbuf -o0 tr '\r' '\n' | 
  while read line2; do echo -en "$l2$c2$line2$el$r0"; done &)
(cat error3 | stdbuf -o0 tr '\r' '\n' | 
  while read line3; do echo -en "$l3$c3$line3$el$r0"; done &)

./prog1 2>error1 | ./prog2  2>error2 | ./prog3 2>error3

wait

rm error{1..3}      # remove named pipes

tput rc             # put cursor below taskbars to finish gracefully
echo
echo "Test finished"

We added colors, different for each line of the taskbar, with strings produced by tput.

Enjoy.