Chapter 4 Automate routine operation and maintenance activities
System administrators have a wide variety of administrative activities that need to be performed. These activities may include file handling, logging, managing CPU and memory, password handling, and most importantly, taking backups. These activities need to be automated. In this chapter, we will learn to automate these activities using Python.
In this chapter, we will discuss the following topics:
- Receive input via redirection, pipes, and input files
- Runtime password handling in scripts
- Execute external command and get output
- Popup for password input at runtime and authentication
- read configuration file
- Add logging and warning code to the script
- Set caps for CPU and memory usage
- start web browser
- Working with directories and files using the os module
- Create a backup (using rsync)
Receive input via redirection, pipes, and input files
In this part, we'll learn how to let users receive input through redirection, pipes, and external input files.
For receiving redirected input we use stdin. Pipes are another form of redirection. This concept refers to using the output of one program as the input of another program. We can receive input through external files as well as using Python.
input via redirection
stdin and stdout are objects created by the os module. We will write a script to use stdin and stdout.
Create a script called redirection.py and write the following code:
import sys
class Redirection(object):
def __init__(self, in_obj, out_obj):
self.input = in_obj
self.output = out_obj
def read_line(self):
res = self.input.readline()
self.output.write(res)
return res
if __name__ == '__main__':
if not sys.stdin.isatty():
sys.stdin = Redirection(in_obj=sys.stdin, out_obj=sys.stdout)
a = input("Enter a string: ")
b = input("Enter another string: ")
print('Entered strings are:', repr(a), 'and', repr(b))
Run the above program as follows:
$ python3 redirection.py
We will get the following output:
Enter a string: hello Enter another string: python Entered strings are: 'hello' and 'python'
When a program is running in an interactive session, stdin is keyboard input and stdout is the user's terminal. The input() function is used to receive input from the user, and print() is a way to write to the terminal (stdout).
piped into
Pipes are another form of redirection. This technique is used to pass information from one program to another. The symbol | indicates a pipe. By using pipelining, we can use more than two commands, taking the output of one command as input to the next command.
Let's take a look at how to use pipes to receive input. First we'll write a simple script that returns divisibility down. Create a script called accept_by_pipe.py and add the following code:
import sys
for n in sys.stdin:
print(int(n.strip())//2)
Running the script, we will get the following output:
$ echo 15 | python3 accept_by_pipe.py Output result: 7
In the above script, stdin is a keyboard input. We perform a downward division of the number entered at runtime. Dividing down returns only the integer part of the quotient. When running the program, we passed 15 through the pipe symbol | , followed by our script name. So we put 15 as input to the script. So division down is performed and we get 7 as the result.
We can pass multiple inputs to the script. In the next execution, we pass in several input values: 15, 45 and 20. To handle multiple input values, we wrote a for loop in the script. So first take 15 as input, then 45, then 20. The output for each input will be printed on a new line because we put \n between the input values. To enable interpretation of backslashes, we pass the -e flag:
$ echo -e '15\n45\n20' | python3 accept_by_pipe.py Output result: 7 22 10
After doing this run, we get the results 7, 22 and 10 which are divisible down by 15, 45 and 20 respectively, each on a new line.
Input via input file
In this part, we will learn how to receive input from an input file. It is easier to accept an input file as input in Python. We'll look at it with an example. But first create a simple text file called sample.txt and write the following code:
# sample.txt Hello World Hello Python
Let's create a script called accept_by_input_file.py and write the following code:
i = open('sample.txt', 'r')
o = open('sample_output.txt', 'w')
a = i.read()
o.write(a)
Running the program, we will get the following output:
$ python3 accept_by_input_file.py $ cat sample_output.txt Hello World Hello Python
Runtime password handling in scripts
In this section, we look at a simple example of handling passwords in a script. We will create a script called handling_password.py and write the following:
import sys
import paramiko
import time
ip_address = "192.168.2.106"
username = "student"
password = "training"
ssh_client = paramiko.SSHClient()
ssh_client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh_client.load_system_host_keys()
ssh_client.connect(hostname=ip_address, username=username, password=password)
print("Successful connection", ip_address)
ssh_client.invoke_shell()
remote_connection = ssh_client.exec_command('cd Desktop; mkdir work\n')
remote_connnection = ssh_client.exec_command('mkdir test_folder\n')
# print(remote_connection.read())
ssh_client.close()
Running the above script will result in the following output:
$ python3 handling_password.py output result: Successful connection 192.168.2.106
In the above script, we have used the paramiko module. The paramiko module is a Python implementation of ssh that provides client functionality. The command to install paramiko is as follows:
pip3 install paramiko
In the previous script, we remotely connected to the host 192.168.2.106. The host username and password are provided in the script.
After running the script, on the desktop of 192.168.2.106, two folders, work and test_folder, can be found in the home/ directory of 192.168.2.106.
Translator's note:
1. Please modify the IP, user name, password and other related information by yourself, including the directory entered by cd, please also modify according to the actual situation
2. Solve the warning during operation
# warning message CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding. # Solution pip3 uninstall cryptography pip3 install cryptography==2.4.2 # reason: this is in GitHub is a known issue，The main reason is that the default installed cryptography==2.5 deprecated some API
Execute external command and get output
In this part, we will learn about Python's subprocess module. Using subprocess, it is easy to spawn new processes and get their return codes, execute external commands and start new applications.
Let's see how to execute external commands and get their output through the subprocess module. We will create a script called execute_external_commands.py and write the script as follows:
import subprocess
subprocess.call(["touch", "sample.txt"])
subprocess.call(["ls"])
print("Sample file created")
subprocess.call(["rm", "sample.txt"])
subprocess.call(["ls"])
print("Sample file deleted")
Running the program, we will get the following output:
$ python3 execute_external_commands.py accept_by_input_file.py handling_password.py sample.txt accept_by_pipe.py redirection.py execute_external_commands.py sample_output.txt Sample file created accept_by_input_file.py execute_external_commands.py redirection.py accept_by_pipe.py handling_password.py sample_output.txt Sample file deleted
Capturing output with the subprocess module
In this part, we will learn how to capture the output. We can pass PIPE as an argument to stdout to capture the output. . Write a script named capture_output.py and add the following code:
import subprocess
res = subprocess.run(['ls', '-l'], stdout=subprocess.PIPE,)
print('returncode:', res.returncode)
print('{} bytes in stdout:\n{}'.format(len(res.stdout), res.stdout.decode('utf-8')))
Execute the following command to run the script:
$ python3 capture_output.py
By executing, the following output will be obtained:
returncode: 0 143 bytes in stdout: accept_by_input_file.py accept_by_pipe.py capture_output.py execute_external_commands.py handling_password.py redirection.py sample_output.txt
Translator's Note: Considering the layout and consistency with the original book, the -l parameter in the code was removed at runtime
In the above script, we imported Python's subprocess module, which facilitates output capture. The subprocess module is used to create new processes. It is also useful for connecting input/output pipes and getting return codes. subprocess.run() runs the command passed in as an argument. returncode is the return status of the child process. In the output, if you get a return code of 0, it's a successful run.
Popup for password input at runtime and authentication
In this part we will learn about the getpass module to handle passwords at runtime. The getpass() module in Python pops up to let the user enter a password without printing it. The getpass module is used for user interaction where programs in the terminal need to handle password popups.
Let's look at some examples on how to use the getpass module:
1. Create a script named no_prompt.py and write the following code:
import getpass
try:
p = getpass.getpass()
except Exception as error:
print('ERROR', error)
else:
print('Password entered:', p)
In this script, we do not provide a prompt file for the user. Therefore, the default prompt content is Password.
Run the script as follows:
$ python3 no_prompt.py Password: Password entered: abcd
2. We will add prompt text for password output. Create a script called with_prompt.py and write the following code:
import getpass
try:
p = getpass.getpass("Enter your password:")
except Exception as error:
print('ERROR', error)
else:
print('Password entered:', p)
This way we have written a script with a password input prompt text. The running program is as follows:
$ python3 with_prompt.py Enter your password: Password entered: abcd
Here, we provide the user with a prompt to Enter your password.
Now let's write a script that will print a normal message if the password is entered incorrectly, without prompting for the correct password.
3. Write a script named getpass_example.py and add the following code:
import getpass
passwd = getpass.getpass(prompt='Enter your password:')
if passwd.lower() == '#pythonworld':
print('Welcome!!')
else:
print('The password entered is incorrect!!')
Run the program as follows (here we enter the correct password, ie #pythonworld):
$ python getpass_example.py # Output result: Enter your password: Welcome!!
Check out the message we get if we enter a wrong password:
$ python getpass_example.py # output result Enter your password: The password entered is incorrect!!
The script we wrote here will not ask to re-enter the password when the password is entered incorrectly.
Next, let's write a script that requires re-entering the correct password when the password is entered incorrectly. To get the logged in user, we use getuser(). The getuser() function will return the system logged in user. Create a script named password_prompt_again.py and write the following code:
import getpass
user_name = getpass.getuser()
print('User Name : %s' % user_name)
while True:
passwd = getpass.getpass('Enter your password : ')
if passwd == '#pythonworld':
print('Welcome!!!')
break
else:
print('The password you entered is incorrect.')
Run the program, you will get the following output:
$ python3 password_prompt_again.py User Name : student Enter your password : The password you entered is incorrect. Enter your password : Welcome!!!
read configuration file
In this part we will learn about the configparser module in Python. By using the configparser module, we can manage user-editable configuration files for our application.
These configuration files are often used by users or system administrators to edit the file to set some default values of the application through a common text editor, and then the application will read and parse the file, and run according to the content written in the file.
To read configuration files, use the read() method in configparser. Below we write a normal script called read_config_file.py. Before doing this, write an .ini file named read_simple.ini and add the following content:
[bug_tracker] url = https://www.baidu.com/
Create read_config_file.py and add the following:
from configparser import ConfigParser
p = ConfigParser()
p.read('read_simple.ini')
print(p.get('bug_tracker', 'url'))
Running read_config_file.py, we will get the following output:
$ python3 read_config_file.py # Output result: https://www.baidu.com/
The read() method can accept more than one filename. When each filename is scanned and the file exists, it is opened and read. Below we write a script to read more than one filename. Create a script called read_many_config_file.py and write the following code:
from configparser import ConfigParser
import glob
p = ConfigParser()
files = ['hello.ini', 'bye.ini', 'read_simple.ini', 'welcome.ini']
files_found = p.read(files)
files_missing = set(files) - set(files_found)
print('Files found: ', sorted(files_found))
print('Files missing: ', sorted(files_missing))
Running the above script, we will get the following output:
$ python3 read_many_config_file.py # Output result: Files found: ['read_simple.ini'] Files missing: ['bye.ini', 'hello.ini', 'welcome.ini']
In the above example, we have used the configparser module in Python which helps in managing configuration files. First we create a list called files. The read() function will read the configuration file. In this example, we have created a variable called files_found to store the names of the configuration files present in the directory. Next we create another variable called files_missing that returns the names of files that do not exist in the directory. Finally, we print out the filenames that exist and do not exist.
Add logging and warning code to the script
In this part we learn about the logging and alerting modules in Python. The logging module will log events that occur in the program. warnings warn programmers about changes in the language and libraries.
Let's look at a simple log example. We will write a script called logging_example.py and write the following code:
import logging
LOG_FILENAME = 'log.txt'
logging.basicConfig(filename=LOG_FILENAME, level=logging.DEBUG,)
logging.debug('This message should go to the log file.')
with open(LOG_FILENAME, 'rt') as f:
prg = f.read()
print('FILE:')
print(prg)
The running program is as follows:
$ python3 logging_example.py # Output result: FILE: DEBUG:root:This message should go to the log file.
Looking at log.txt, we can see the debugging information printed in the script:
$ cat log.txt # Output result: DEBUG:root:This message should go to the log file.
Below we write a script named logging_warnings_codes.py and add the following code:
import logging
import warnings
logging.basicConfig(level=logging.INFO,)
warnings.warn('This warning is not sent to the logs')
logging.captureWarnings(True)
warnings.warn('This warning is sent to the logs')
Run the script as follows:
$ python3 logging_warnings_codes.py
# Output result:
logging_warnings_codes.py:5: UserWarning: This warning is not sent to the logs
warnings.warn('This warning is not sent to the logs')
WARNING:py.warnings:logging_warnings_codes.py:7: UserWarning: This warning is sent to the logs
warnings.warn('This warning is sent to the logs')
generate warning
warn() is used to generate a warning. Let's look at a simple example that generates a warning. Write a script called generate_warnings.py and add the following code:
import warnings
warnings.simplefilter('error', UserWarning)
print('Before')
warnings.warn('Write your warning message here')
print('after')
Run the script as follows:
python3 generate_warnings.py
# operation result:
Before
Traceback (most recent call last):
File "generate_warnings.py", line 5, in <module>
warnings.warn('Write your warning message here')
UserWarning: Write your warning message here
In the script above, we passed a warning message through warn(). We use a simple filter so that our warnings are handled as error s, which are handled accordingly by the programmer.
Set caps for CPU and memory usage
In this part we will learn how to set CPU and memory usage limits. First write a script that places a CPU usage limit. Create a script called put_cpu_limit.py and add the following code:
import resource
import sys
import signal
import time
def time_expired(n, stack):
print('EXPIRED :', time.ctime())
raise SystemExit('(time ran out)')
signal.signal(signal.SIGXCPU, time_expired)
# Adjust the CPU time limit
soft, hard = resource.getrlimit(resource.RLIMIT_CPU)
print('Soft limit start as :', soft)
resource.setrlimit(resource.RLIMIT_CPU, (10, hard))
soft, hard = resource.getrlimit(resource.RLIMIT_CPU)
print('Soft limit changed to :', soft)
print()
# Consume some CPU time in a pointless exercise
print('Starting:', time.ctime())
for i in range(200000):
for i in range(200000):
v = i * i
# We should never make it this far
print('Exiting :', time.ctime())
Run the above script as follows:
$ python3 put_cpu_limit.py Soft limit start as : -1 Soft limit changed to : 10 Starting: Sun Feb 24 23:57:27 2019 EXPIRED : Sun Feb 24 23:57:37 2019 (time ran out)
Translator's Note: The actual running result may not be 10 seconds. Alan runs the printing time on a local virtual machine with limited resources for 15 seconds, while on the Mac it is exactly 10 seconds.
In the previous script, we used setrlimit() to limit the CPU usage. The limit set in our script is 10 seconds.
start web browser
In this part, we will learn about the webbrowser module in Python. This module contains functions to open URL s in browser applications. Let's look at a simple example. Create a script called open_web.py and add the following code:
import webbrowser
webbrowser.open('https://www.baidu.com')
Run the script as follows:
$ python3 open_web.py The result of the operation is as follows: Url mentioned in open() will be opened in your browser. webbrowser – Command line interface
Translator's Note: The actual test on MacOS does not output relevant information, the command line only outputs True, the command line of the virtual machine is entered, no browser is installed, and there is no return content
We can also use Python's webbrowser module through the command line, and all functions can be used. To use the webbrowser from the command line, run the command as follows:
python3 -m webbrowser -n https://www.baidu.com/
Here, https://www.baidu.com/ will be opened in a browser window. We can use the following two options:
- -n: open a new window
- -t: open a new tab
Working with directories and files using the os module
In this part, we will learn about the os module in Python. Python's os module helps with operating system tasks. To implement these operating system tasks, we need to import the os module.
Let's look at some examples related to handling files and directories:
Create and delete modules
In this section, we'll create a script to look at functions that work with directories on the file system, including creating, listing, and deleting their contents. Create a script named os_dir_example.py and write the following code:
import os
directory_name = 'abcd'
print('Creating', directory_name)
os.makedirs(directory_name)
file_name = os.path.join(directory_name, 'sample_example.txt')
print('Creating', file_name)
with open(file_name, 'wt') as f:
f.write('sample example file')
print('Cleaning up')
os.unlink(file_name)
os.rmdir(directory_name) # Will delete the directory
Run the script as follows:
$ python3 os_dir_example.py # Output result: Creating abcd Creating abcd/sample_example.txt Cleaning up
When creating a directory with mkdir(), all parent directories must have been created. But when using makedirs() to create a directory, any directory will be created, even if the specified path does not exist. unlink() will delete the file path, and rmdir() will delete the directory path.
Check file system contents
In this section, we will use listdir() to list all the contents of a directory. Create a script called list_dir.py and write the following code:
import os import sys print(sorted(os.listdir(sys.argv[1])))
Run the script as follows:
$ python3 list_dir.py /home/student # Output result: ['.bash_history', '.bash_logout', '.bashrc', '.cache', '.local', '.profile', '.python_history', '.ssh', '.viminfo', 'Chapter01', 'Chapter02', 'Chapter03', 'Chapter04', '__pycache__']
By using listdir() we can list all the contents of a folder.
Create a backup (using rsync)
This is the most important job for system operation and maintenance personnel to do. In this part, we will learn to use rsync for backup. The rsync command is used to copy local and remote files and directories, and use rsync to perform data backup. To do this, let's write a script called take_backup.py and write the following code:
import os
import shutil
import time
from sh import rsync
def check_dir(os_dir):
if not os.path.exists(os_dir):
print(os_dir, 'does not exist.')
exit(1)
def ask_for_confirm():
ans = input('Do you want to Continue? yes/no\n')
global con_exit
if ans == 'yes':
con_exit = 0
return con_exit
elif ans == 'no':
con_exit = 1
return con_exit
else:
print('Answer with yes or no.')
ask_for_confirm()
def delete_files(ending):
for r, d, f in os.walk(backup_dir):
for files in f:
if files.endswith('.' + ending):
os.remove(os.path.join(r, files))
backup_dir = input('Enter directory to backup\n') # Enter directory name
check_dir(backup_dir)
print(backup_dir, 'saved.')
time.sleep(3)
backup_to_dir = input('Where to backup?\n')
check_dir(backup_to_dir)
print('Doing the backup now!')
ask_for_confirm()
if con_exit == 1:
print('Aborting the backup process!')
exit(1)
rsync('-auhv', '--delete', '--exclude=list+found', '--exclude=/sys', '--exclude=/tmp', '--exclude=/proc', '--exclude=/mnt', '--exclude=/dev', '--exclude=/backup', backup_dir, backup_to_dir)
The result of running the script is as follows:
$ python3 take_backup.py Enter directory to backup /home/student/work /home/student/work saved. Where to backup? /home/student/Desktop Doing the backup now! Do you want to Continue? yes/no yes
Translator's note:
1. If it prompts ImportError: No module named ‘sh’, please execute pip3 install sh
2. The Deskstop directory needs to be created by itself
Now look in the Desktop/ directory, you will find the work folder in that directory. rsync uses some options, as follows:
- -a: archive
- -u: upgrade
- -h: human-readable format
- -v: Verbose information (verbose)
- --delete: delete irrelevant files on the receiving end
- --exclude: queuing discipline
Summarize
In this chapter we learned how to automate routine execution tasks. Learned different techniques for receiving input: prompting for passwords at runtime, executing external commands, reading configuration files, adding warnings to scripts, launching browsers from scripts and the command line, handling files and directories using the os module, and making backups .
In the next chapter, we will learn about os and data processing. We will also learn about the tarfile module and how to use it.
☞☞☞ Chapter V Files, Directories and Data Processing
After-school questions
- How to use the readline module?
- What are the Linux commands to read, create new files, delete files, and list files in the current directory?
- What packages are available in Python to run Linux/Windows commands?
- How to read or set new values for configuration file init?
- List libraries that can be used to view CPU usage?
- List the different ways to receive input from the user?
- What is the difference between sort and sorted?
further reading
- Basic commands of Linux: https://maker.pro/linux/tutorial/basic-linux-commands-for-beginners
- Selenium Web Driver Documentation: https://selenium-python.readthedocs.io/index.html