In this article, learn how to execute Shell commands (such as ls, mkdir, or grep) in a Golang program, how to pass I/O to running commands through stdin and stdout, and manage long-running commands. For a better understanding, several examples are provided for different scenarios from shallow to deep, and I hope you can understand it easily.

exec package

Use the official os/exec package to execute external commands. When you execute shell commands, you need to run code outside the Go application, so these commands need to be run in subprocesses. As shown below:

Each command runs as a subprocess in the Go application and exposes stdin and stdout attributes, which we can use to read and write process data.

Run basic Shell commands

Run a simple command and read data from its output by creating an instance of *exec.Cmd. In the following example, ls is used to list the files in the current directory and its output is printed from the code:

// create a new *Cmd instance
// here we pass the command as the first argument and the arguments to pass to the command as the
// remaining arguments in the function
cmd := exec.Command("ls", "./")

// The `Output` method executes the command and
// collects the output, returning its value
out, err := cmd.Output()
if err != nil {
  // if there was any error, print it here
  fmt.Println("could not run command: ", err)
}
// otherwise, print the output from running the command
fmt.Println("Output: ", string(out))

Because the program is run in the current directory, the files in the root directory of the project are output:

> go run shellcommands/main.go

Output:  LICENSE
README.md
command.go

When running exec, the program does not spawn a shell, but runs the given command directly, which means that no shell-based processing, such as glob patterns or expansion, is done. For example, when running the ls ./*.md command, it does not output readme.md as if we run the command in that shell.

Execute long-running commands

The preceding example executes the ls command and returns the result immediately, but what happens when the command output is continuous or takes a long time to execute? For example, running the ping command will periodically obtain continuous results:

ping www.baidu.com 
PING www.a.shifen.com (36.152.44.95) 56(84) bytes of data.
64 bytes from 36.152.44.95 (36.152.44.95): icmp_seq=1 ttl=128 time=11.1 ms
64 bytes from 36.152.44.95 (36.152.44.95): icmp_seq=2 ttl=128 time=58.8 ms
64 bytes from 36.152.44.95 (36.152.44.95): icmp_seq=3 ttl=128 time=28.2 ms
64 bytes from 36.152.44.95 (36.152.44.95): icmp_seq=4 ttl=128 time=11.1 ms
64 bytes from 36.152.44.95 (36.152.44.95): icmp_seq=5 ttl=128 time=11.5 ms
64 bytes from 36.152.44.95 (36.152.44.95): icmp_seq=6 ttl=128 time=53.6 ms
64 bytes from 36.152.44.95 (36.152.44.95): icmp_seq=7 ttl=128 time=10.2 ms
64 bytes from 36.152.44.95 (36.152.44.95): icmp_seq=8 ttl=128 time=10.4 ms
64 bytes from 36.152.44.95 (36.152.44.95): icmp_seq=9 ttl=128 time=15.8 ms
64 bytes from 36.152.44.95 (36.152.44.95): icmp_seq=10 ttl=128 time=16.5 ms
64 bytes from 36.152.44.95 (36.152.44.95): icmp_seq=11 ttl=128 time=10.9 ms
^C64 bytes from 36.152.44.95: icmp_seq=12 ttl=128 time=9.92 ms

If you try to execute this kind of command using cmd.Output, you will not get any results, because the Output method waits for the end of command execution, while ping executes indefinitely. Therefore, you need to customize the Stdout property to read continuous output:

cmd := exec.Command("ping", "google.com")

// pipe the commands output to the applications
// standard output
cmd.Stdout = os.Stdout

// Run still runs the command and waits for completion
// but the output is instantly piped to Stdout
if err := cmd.Run(); err != nil {
  fmt.Println("could not run command: ", err)
}

Run the program again, and the output is similar to that executed in the Shell.

By directly assigning the Stdout property, we can capture output throughout the command lifecycle and process it as soon as it is received. The io interaction between processes is shown in the following figure:

custom write output

Instead of using os.Stdout, custom write output can also be created by implementing the io.Writer interface.

The following custom code adds a "received output: " prefix to each output block:

type customOutput struct{}

func (c customOutput) Write(p []byte) (int, error) {
	fmt.Println("received output: ", string(p))
	return len(p), nil
}

Now assign a custom write output instance to the command output:

cmd.Stdout = customOutput{}

Run the program again and you will get the following output.

Use Stdin to pass input to commands

The preceding example does not give any input to the command (or provides limited input as a parameter), and in most scenarios the input information is passed through the Stdin stream. A typical example is the grep command, which can be piped from one command string to another:

➜  ~ echo "1. pear\n2. grapes\n3. apple\n4. banana\n" | grep apple
3. apple

Here the output of echo is passed to grep as stdin, a group of fruits is input, and only apple is output through grep filtering.

The *Cmd instance provides an input stream for writing, and the following example uses it to pass input to the grep subprocess:

cmd := exec.Command("grep", "apple")

// Create a new pipe, which gives us a reader/writer pair
reader, writer := io.Pipe()
// assign the reader to Stdin for the command
cmd.Stdin = reader
// the output is printed to the console
cmd.Stdout = os.Stdout

go func() {
  defer writer.Close()
  // the writer is connected to the reader via the pipe
  // so all data written here is passed on to the commands
  // standard input
  writer.Write([]byte("1. pear\n"))
  writer.Write([]byte("2. grapes\n"))
  writer.Write([]byte("3. apple\n"))
  writer.Write([]byte("4. banana\n"))
}()

if err := cmd.Run(); err != nil {
  fmt.Println("could not run command: ", err)
}

Output result:

3. apple

end child process

There are some commands that run indefinitely and need to be able to signal to end. For example, if you run a web service with python3 -m http.server or sleep 10000, the child process will run for a long time or run indefinitely.

To stop a process, a kill signal needs to be sent from the application, which can be achieved by adding a context instance to the command. If the context is cancelled, the command will also terminate execution:

ctx := context.Background()
// The context now times out after 1 second
// alternately, we can call `cancel()` to terminate immediately
ctx, _ = context.WithTimeout(ctx, 1*time.Second)

// sleep 10 second 
cmd := exec.CommandContext(ctx, "sleep", "10")

out, err := cmd.Output()
if err != nil {
  fmt.Println("could not run command: ", err)
}
fmt.Println("Output: ", string(out))

Run the program and output the result after 1 second:

could not run command:  signal: killed
Output:  

When the command needs to be run within a limited time or the command does not return a result within a certain period of time, the backup logic is executed.

Summarize

So far, we have learned various ways to execute and interact with unix shell commands. Here are some things to be aware of when using the os/exec package:

• Use cmd.Output when you wish to execute simple commands that usually don't provide much output
• For functions with continuous or long output you should use cmd.Run and interact with it via cmd.Stdout and cmd.Stdin
• In a production scenario, if the process does not respond within a given time, it must have a timeout and end function, and you can use the cancel context to send a termination command