According to the blog to learn linux system programming notes, from https://tennysonsky.blog.csdn.net/
1.Linux system programming
There are three ways to make system calls to the Linux operating kernel
- Shell, the system call of operating kernel by shell interpreter through shell command
- Library function: the user operates the system call of the kernel through the interface of the application layer library function, such as fread
- The application layer system call can directly call the system of the kernel
2. File IO
2.1 file descriptor
The file descriptor is a non negative integer. When opening an existing file or a new file, the kernel will return a file descriptor, which is used to specify the open file.
In the system call (file IO), the file descriptor plays the role of identifying the file. If you want to operate the file, it is to operate the file descriptor
When a program runs or a process starts, the system will automatically create three file descriptors
#define STDIN_ Fileno 0 / / standard input file descriptor #define STDOUT_ Fileno 1 / / file descriptor of standard output #define STDERR_ Fileno 2 / / standard error file descriptor
If you open the file yourself, the file descriptor will be returned, and the file descriptor is generally created from small to large
In Linux, a process can only open NR at most_ OPEN_ Default (i.e. 1024 files) (windos 8191), so when the file is no longer used, the close() function should be called in time to close the file.
2.2 open
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
/*
pathname:Path flags:open behavior flag bit mode: permission
*/
//Use when file exists
int open(const char *pathname, int flags);
//Use when file does not exist
int open(const char *pathname, int flags, mode_t mode);
int creat(const char *pathname, mode_t mode);
int openat(int dirfd, const char *pathname, int flags);
int openat(int dirfd, const char *pathname, int flags, mode_t mode);
2.3 perror
#include <stdio.h> #include <errno.h> //The error message of the failure of the output function call will splice the error message after the string you enter void perror(const char *s); const char * const sys_errlist[]; int sys_nerr; //Error is a global variable through which the error code is obtained int errno;
cat /usr/include/asm-generic/errno-base.h //Query the definition of error code message
#define EPERM 1 /* Operation not permitted */ #define ENOENT 2 /* No such file or directory */ #define ESRCH 3 /* No such process */ #define EINTR 4 /* Interrupted system call */ #define EIO 5 /* I/O error */ #define ENXIO 6 /* No such device or address */ #define E2BIG 7 /* Argument list too long */ #define ENOEXEC 8 /* Exec format error */ #define EBADF 9 /* Bad file number */ #define ECHILD 10 /* No child processes */ #define EAGAIN 11 /* Try again */ #define ENOMEM 12 /* Out of memory */ #define EACCES 13 /* Permission denied */ #define EFAULT 14 /* Bad address */ #define ENOTBLK 15 /* Block device required */ #define EBUSY 16 /* Device or resource busy */ #define EEXIST 17 /* File exists */ #define EXDEV 18 /* Cross-device link */ #define ENODEV 19 /* No such device */ #define ENOTDIR 20 /* Not a directory */ #define EISDIR 21 /* Is a directory */ #define EINVAL 22 /* Invalid argument */ #define ENFILE 23 /* File table overflow */ #define EMFILE 24 /* Too many open files */ #define ENOTTY 25 /* Not a typewriter */ #define ETXTBSY 26 /* Text file busy */ #define EFBIG 27 /* File too large */ #define ENOSPC 28 /* No space left on device */ #define ESPIPE 29 /* Illegal seek */ #define EROFS 30 /* Read-only file system */ #define EMLINK 31 /* Too many links */ #define EPIPE 32 /* Broken pipe */ #define EDOM 33 /* Math argument out of domain of func */ #define ERANGE 34 /* Math result not representable */
2.4 close
Success returns 0, failure returns - 1
#include <unistd.h> //Close file descriptor int close(int fd);
2.5 write
#include <unistd.h> //fd: file descriptor //addr: data header address //count: the length (bytes) of the written data. Generally, the number of data can be written into the file, no more or less ssize_t write(int fd, const void *buf, size_t count);
Return value success: the number of bytes actually written to the data
Failed: - 1
2.6 read
#include <unistd.h> //fd: file descriptor //addr: first memory address //count: number of bytes read ssize_t read(int fd, void *buf, size_t count);
Return value success: the number of bytes of data actually read
Failed: - 1
Note that reading the end of the file returns 0
2.7 remove
#include <stdio.h> //delete int remove(const char *pathname);
2.8 system calls and library functions
2.8.1 no system call is required
You don't need to switch to kernel space to complete all functions, such as strcpy bzero and other string operation functions
2.8.1 system call required
It is necessary to switch to kernel space. Such functions can realize corresponding functions by encapsulating system calls
2.8.1 relationship between library function and system call
Not all system calls are encapsulated into library functions, and many functions must be realized through system calls
The system call takes time. The CPU is working in the kernel state. When the system call occurs, the user state, stack and memory environment need to be saved
Then switch to kernel mode
After that, you have to switch back to user mode, which will consume a lot of time
When library functions access files, different types of buffers are set according to needs, which directly reduces the number of IO system calls.
In fact, most library functions also call the system call, but set the buffer
3. Process
3.1 definitions
Program: an executable file of a program stored on a storage medium
Process: a process is an execution instance of a program, including program counters, registers, and the values of current variables. The program is static and the process is dynamic
A program is an ordered collection of instructions. A process is a process of program execution. The state of a process is changing, including the creation, scheduling and extinction of a process
In the Linux operating system, a task is completed through a process, process It is the basic unit of managing transactions. The process has its own independent processing environment
And system resources (processors, memory, I/O devices, data, programs)
3.2 status
Ready state:
The process has all the conditions for execution and is waiting to allocate CPU processing time.
Execution status:
The process is running on CPU.
Waiting state:
The state in which a process cannot be continued temporarily because it does not meet some execution conditions.
Both the ready state and the waiting state are not executed, but they are different. The ready state means that the conditions are met. Before the time is up, the waiting state does not meet the conditions.
Similarly, these three states of a process can be converted to each other
Execution status – > waiting status:
When an executing process cannot continue to execute because it waits for an event to occur, it changes from execution state to waiting state
Waiting state – > ready state:
A process in a waiting state changes from a waiting state to a ready state if its waiting event occurs
Ready status – > execution status:
When the cpu time slice that the ready state process is waiting for arrives, the process will change from ready state to execution state
Execution status – > ready status:
In the process of execution, the process in the execution state has to give up the cpu because a time slice allocated to it has run out, so the process changes from the execution state to the ready state
3.3 process control block (PCB)
When we run a program to make it a process, the system will open up a memory space to store the data information related to the process, which is through the structure task_struct recorded
Process control block is the most important record data structure in operating system. The process control block records all the information needed to describe the progress of the process and control the operation of the process. It is the only sign of the existence of the process.
There are a lot of information in the process control block, of which the more important is the process number. As for other information, we will not discuss it in detail here.
[the external chain picture transfer fails. The source station may have an anti-theft chain mechanism. It is recommended to save the picture and upload it directly (img-ZZNQiSD9-1649984831137)(LinuxSysCode.assets/20150624100139840.png)]
3.3 process number
Each process is identified by a process number, and its type is pid_t (unsigned integer), process number range: 0 ~ 32767. The * * process number is always unique, but the process number can be reused** When a process terminates, its process number can be used again
The system allows a process to create a new process. The new process is a sub process. The sub process can also create a new sub process to form a process tree structure model.
All processes of the whole Linux system are also a tree structure.
The tree root is automatically constructed by the system, that is, process 0 executed in the kernel state. It is the ancestor of all processes. A process with process number 0 is usually a scheduling process, which is often called a swap process. Process 1 (kernel state) is created by process 0.
No. 1 is responsible for performing some initialization of the kernel and system configuration, and creating several kernel threads for cache and virtual main memory management. Then * *, process 1 calls execve() to run the executable program init, and evolves into user state process 1, that is, init process**
The init process is the ancestor of all processes. Except for the scheduling process No. 0, all processes are created directly or indirectly by the init process
Three different process numbers
Process number (PID):
A nonnegative integer number that identifies the process.
Parent process number (PPID):
Any process (except init process) is created by another process, which is called the parent process of the created process, and the corresponding process is called the parent process number (PPID). For example, process A creates process B, and the process number of process A is the parent process number of process B.
Process group number (PGID):
A process group is a collection of one or more processes. They are interrelated. The process group can receive various signals from the same terminal, and the associated process has a process group number (PGID). This process is a bit similar to QQ group. The group is equivalent to QQ group, and each process is equivalent to each friend. Pulling each friend into this QQ group is mainly to facilitate management, especially when notifying something, as long as you shout in the group, everyone will receive it, which is simple and rough. However, this process group number is somewhat different from the QQ group number. By default, the current process number will be the current process group number.
3.4 geipid() getppid() getpgid()
#include <sys/types.h>
#include <unistd.h>
pid_t getpid(void); //Get this process number (PID)
pid_t getppid(void); //Gets the parent process number (PPID) of the process calling this function
pid_t getpgid(pid_tpid); //Get process group number (PGID)
3.5 fork
#include <sys/types.h> #include <unistd.h> //It is used to create a new process from an existing process. The new process is called a child process and the original process is called a parent process. pid_t fork(void);
Return value:
Success: 0 is returned in the child process, and the child process ID is returned in the parent process. pid_t. Is an unsigned integer.
Failure: Return - 1
The two main reasons for failure are:
(1) The current number of processes has reached the upper limit specified by the system. At this time, the value of errno is set to EAGAIN.
(2) The system is out of memory, and the value of errno is set to ENOMEM.
If the code area of the parent-child process is not distinguished, the same instruction will be run twice by the parent-child process
In short, after a process calls the fork() function, the system first allocates resources to the new process, such as the space to store data and code. Then copy all the values of the original process to the new process, and only a few values are different from those of the original process. It's equivalent to cloning yourself.
In fact, more accurately, the use of fork() in Linux is realized through copy on write (COW Technology). Copy on write is a technology that can delay or even avoid copying data. At this time, the kernel does not copy the address space of the whole process, but allows the parent and child processes to share the same address space. The address space is copied only when it needs to be written, so that each has its own address space. In other words, resources are copied only when they need to be written. Before that, they can only be shared in a read-only manner.
You can simply think that the code of the parent-child process is the same. In this case, what the parent process does and what the child process does (as in the above example) cannot meet our requirements for multitasking. We need to find a way to distinguish the parent-child process, which is through the return value of fork().
Distinguish parent-child processes by return value
It should be noted that in the address space of the child process, the child process starts executing code after the fork() function.
Generally speaking, it is uncertain whether the parent process will execute first or the child process will execute first after fork(). This depends on the scheduling algorithm used by the kernel
Do not assume that the child process will not proceed until the parent process has finished executing
In the following example, to verify that the address spaces of parent and child processes are independent
It is known that modifying the values of variables A and b in the child process does not affect the values of parent processes a and b.
However, the child process will inherit some public space of the parent process, such as disk space and kernel space
The offset of the file descriptor is stored in kernel space, so if the parent process changes the offset, the offset obtained by the child process is changed
3.6 process suspension (sleep)
If a process does not operate within a certain period of time, it is called process suspension, which is to turn the process to the waiting state
3.7 process waiting
3.7.1 wait
Sometimes simple inter process synchronization is required between parent and child processes, such as the parent process waiting for the child process to end
#include <sys/types.h> #include <sys/wait.h> //Status: status information when the process exits. pid_t wait(int *wstatus);
Parameters:
If the value of the parameter status is not NULL, wait() will take out and store the state of the child process when it exits. This is an integer value (int), indicating whether the child process exits normally or ends abnormally.
This exit information contains multiple fields in an int. it is meaningless to use this value directly. We need to use macro definition to get each field.
Return value:
Success: the process number of the child process has ended
Failed: - 1
Let's learn about the two most commonly used macro definitions and get the exit information of the child process:
WIFEXITED(status)
If the child process terminates normally, the field value taken out is non-zero.
WEXITSTATUS(status)
Returns the exit status of the child process. The exit status is saved in 8 ~ 16 bits of the status variable. Before using this macro, you should first judge whether the subprocess exits normally with the macro disabled, and you can use this macro only after the subprocess exits normally.
In essence, the functions of system calls waitpid() and wait() are exactly the same, but waitpid() has two more parameters pid and options that can be controlled by users, which provides us with another more flexible way of programming.
3.7.2 waitpid
Wait for the child process to terminate. If the child process terminates, the resources of the child process will be recycled
//pid > 0 waits for a child process whose process ID is equal to pid. //pid = 0 waits for any child process in the same process group. If the child process has joined another process group, waitpid will not wait for it. //pid = -1 wait for any child process. At this time, waitpid works the same as wait. //pid < - 1 waits for any child process in the specified process group whose ID is equal to the absolute value of pid. //options: options provides some additional options to control waitpid(). //0: the same as wait(), blocking the parent process and waiting for the child process to exit. //WNOHANG； If there are no child processes that have ended, return immediately. //WUNTRACED: if the child process is suspended, this function returns immediately and ignores the end state of the child process. (because it involves some knowledge of tracking and debugging, it is rarely used and not written) -------- pid_t waitpid(pid_t pid, int *wstatus, int options);
Return value
There are three situations:
When it returns normally, waitpid() returns the collected process number of the child process;
If the option WNOHANG is set, and waitpid() in the call finds that there are no child processes that have exited to wait, it returns 0;
If there is an error in the call, it will return - 1. At this time, errno will be set to the corresponding value to indicate the error. For example, when the child process corresponding to pid does not exist, or this process exists but is not the child process of the calling process, waitpid() will return an error. At this time, errno is set to ECHILD;
3.8 process termination (exit,_exit)
exit is a library function_ exit is a system call
#include <unistd.h> void _exit(int status);
continue: end this cycle
break: jump out of the entire loop, or jump out of the switch() statement
return: end the current function
We can use exit() or_ Exit() to end the current process.
3.9 process exit cleanup (atexit)
#include <stdlib.h> //The function process that is called before the top note of the registration process is terminated and the registration function is executed. int atexit(void (*function)(void));
0 returned successfully
3.10 process creation vfork
#include <sys/types.h> #include <unistd.h> pid_t vfork(void);
Success: 0 is returned in the child process, and the child process ID is returned in the parent process. pid_t. Is an unsigned integer.
Failed: Return - 1.
fork() and vfock() both create a process. What's the difference between them?
1) fork(): the execution order of parent-child processes is uncertain.
vfork(): ensure that the child process runs first, and the parent process may not be scheduled to run until it calls exec (process replacement) or exit (exit process).
2) fork(): the child process copies the address space of the parent process. The child process is a copy of the parent process.
vfork(): the child process shares the address space of the parent process (to be exact, the data is shared with the parent process before calling exec (process replacement) or exit (exit process))
The child process of exec will have its own process space
Run first
Shared space
3.11 process replacement
3.11.1 introduction
Under the Windows platform, we can run the executable program by double clicking to make the executable program become a process; On the Linux platform, we can/ Run, let an executable program become a process.
If we are already running a program (process), how to start an external program inside the process, and the kernel reads the external program into memory to make it execute into a process? Here we implement it through the exec function family
The exec function family provides six ways to start another program in a process. The function of exec family is to find the executable file according to the specified file name or directory name and use it to replace the contents of the calling process. In other words, it is to execute an executable file inside the calling process.
When a process calls an exec function * *, the process is completely replaced by a new program, * * and the new program is executed from its main function. Because calling exec does not create a new process, the process ID before and after (of course, the parent process number, process group number, current working directory...) has not changed. Exec just replaces the body, data, heap and stack segments of the current process with another new program (process replacement).
3.11.2 function family
l(list): parameter address list, ending with null pointer.
v(vector): the address of the pointer array containing the address of each parameter.
p(path): search for executable files according to the directory specified by the PATH environment variable.
e(environment): the address of the pointer array containing the string address of the environment variable.
#include <unistd.h>
int execl(const char *path, const char *arg, .../* (char *) NULL */);
int execlp(const char *file, const char *arg, .../* (char *) NULL */);
int execle(const char *path, const char *arg, .../*, (char *) NULL, char * const envp[] */);
int execv(const char *path, char *const argv[]);
int execvp(const char *file, char *const argv[]);
int execvpe(const char *file, char *const argv[],
char *const envp[]);
The exec function family is different from ordinary functions. The functions in the exec function family will not return after successful execution, and the code below the exec function family cannot be executed. Only when the call fails, they will return - 1, and then execute down from the call point of the original program.
3.11.3 execl
3.11.4 execv
3.11.5 execlp execvp
The difference between execlp() and execl() is that the executable program specified by execlp() can not take the PATH name. If it does not take the PATH name, it will look for the executable program in the directory specified by the environment variable PATH, while the executable program specified by execl() must take the PATH name.
3.11.6 execle execve
What execle() and execve() change is the environment variable of the program started by exec (it will only change the environment variable of the process and will not affect the environment variable of the system). The programs started by the other four functions use the default system environment variable.
3.12 special process
3.12.1 zombie process
The process has finished running, but the resources occupied by the process have not been recycled. Such a process is called a zombie process.
When each process exits, the kernel releases all the resources of the process, including open files, occupied memory, etc. However, some information * * is still reserved for it, which mainly refers to the information of the process control block (including process number, exit status, running time, etc.) *.
This will lead to a problem. If the process does not call wait() or waitpid(), the retained information will not be released, and its process number will always be occupied, but the process number that the system can use is limited. If a large number of dead processes are generated, the system will not be able to generate new processes because there are no available process numbers
3.12.2 orphan process
A child process whose parent process has finished running but whose child process is still running (not finished running) is called Orphan process (Orphan Process). Orphan processes will eventually be adopted by the init process (process number 1), and the init process will complete the status collection for them.
Orphan process is a process without a parent process. In order to prevent the orphan process from becoming a zombie process because it cannot release the occupied resources when it exits, init process with process number 1 will accept these orphan processes. This process is also called "adoption"
The init process will wait() its exited child process cyclically. In this way, when an orphan process sadly ends its life cycle, init process will deal with all its aftermath on behalf of the party and the government. Therefore, the orphan process will not do any harm.
3.12.3 (wizard) daemon
Daemon Process, which is commonly referred to as Daemon Process, is a background service process in Linux. It is a long-lived process, usually independent of the control terminal, and periodically performs some tasks or waits to deal with some events.
The daemon is a special orphan process. This process is separated from the terminal. Why should it be separated from the terminal? The reason why it is separated from the terminal is to avoid the process being interrupted by the information generated by any terminal, and the information in the execution process is not displayed on any terminal. In Linux, the interface between each system and the user is called the terminal. Every process running from the terminal will be attached to the terminal. This terminal is called the control terminal of these processes. When the control terminal is closed, the corresponding process will be closed automatically.
Most Linux servers are implemented using daemons. For example, Internet server inetd, Web server httpd, etc
How to view Daemons
At the terminal: ps axj
- a means to list not only the processes of the current user, but also the processes of all other users
- x indicates that not only processes with control terminals but also all processes without control terminals are listed
- j means to list information related to job control
From the above figure, we can see some characteristics of guard:
- Daemons are basically started as super users (UID 0)
- No control terminal (TTY is?)
- Terminal process group ID is - 1 (TPGID indicates terminal process group ID)
Generally, the daemon can be started in the following ways:
- When the system starts, it is started by startup scripts, which are usually placed in / etc / RC D directory;
- Use inetd super server to start, such as telnet;
- Processes that are started regularly by cron and by nohup on the terminal are also daemons.
4. Interprocess communication (signal)
4.1 General
A process is an independent resource allocation unit. Resources between different processes (processes mentioned here usually refer to user processes) are independent and have no association. Resources of another process (such as open file descriptors) cannot be accessed directly in one process.
**However, * * processes are not isolated. Different processes need information interaction and state transmission
Therefore, Inter Processes Communication (IPC) is required.
Purpose of interprocess communication:
- Data transfer: one process needs to send its data to another process.
- Notification event: a process needs to send a message to another process or group of processes to notify it (them) of an event (such as notifying the parent process when the process terminates).
- Resource sharing: multiple processes share the same resources. In order to do this, the kernel needs to provide mutual exclusion and synchronization mechanisms.
- Process control: some processes want to fully control the execution of another process (such as Debug process). At this time, the control process wants to be able to intercept all traps and exceptions of another process and know its state changes in time.
The main communication mechanisms of interprocess communication supported by Linux operating system:
The essence of interprocess communication: as long as the system creates a process, it will allocate 4G virtual memory to the current process
4G virtual memory is divided into 0-3G user space and (3-4G) kernel space. The user space is private to the process.
The heap area, stack area, data area and code area mentioned earlier are the user's own space
Kernel space is shared by all processes, so the basic way of communication between most processes is actually the operation of kernel space
4.2 signal overview
4.2.1 introduction
Semaphores are the oldest means of communication between Linux processes. The signal is Software interrupt , it is a simulation of interrupt mechanism at the software level asynchronous communication The way. Signals can cause a running process to be interrupted by another running asynchronous process to deal with an emergency
The signal can directly interact with the user space process and the kernel space process, and the kernel process can use it to inform the user space process of what system events have occurred.
A complete signal cycle includes three parts: signal generation, signal registration in the process, signal cancellation in the process, and execution of signal processing function. As shown in the figure below:
Linux can use the command: kill -l ("l" is a letter) to view the corresponding signal.
In the list, the signals numbered 1 ~ 31 are the signals supported by traditional UNIX, which are unreliable signals (non real-time), and the signals numbered 32 ~ 63 are later expanded, which are called reliable signals (real-time signals). The difference between unreliable signals and reliable signals is that the former does not support queuing, which may cause signal loss, while the latter does not. Unreliable signals generally have definite purposes and meanings, while reliable signals can be customized by users. For more details, see the Linux signal list.
4.2.2 signal generation mode
1)When the user presses some terminal keys, a signal will be generated. Press on the terminal“ Ctrl+c"Key combinations usually produce interrupt signals SIGINT，Press on the terminal“ Ctrl+\"The key usually generates an interrupt signal SIGQUIT，Press on the terminal“ Ctrl+z"The key usually generates an interrupt signal SIGSTOP Wait. 2)A hardware exception will generate a signal. Divisor 0, invalid memory access, etc. These conditions are usually detected by the hardware and notified to the kernel, and then the kernel generates appropriate signals to send to the corresponding process. 3)Software exceptions will generate signals. A signal is generated when a software condition is detected and notified to the relevant process. 4)call kill() Function will send a signal. Note: the owners of the receive signal process and the send signal process must be the same, or the owner of the send signal process must be a superuser. 5)function kill The command will send a signal. This program is actually using kill Function to send a signal. This command is also commonly used to terminate a runaway background process.
4.3 kill
#include <sys/types.h> #include <signal.h> //Sends a signal to the specified process. //When using the kill() function to send a signal, the owners of the receiving process and the sending process must be the same, or the owner of the sending process is a super user. //pid > 0: send the signal to the process with process ID pid. //pid = 0: send the signal to all processes in the process group where the current process is located. //pid = -1: send the signal to all processes in the system. //pid < - 1: send the signal to all processes of the specified process group. This process group number is equal to the absolute value of pid. int kill(pid_t pid, intsignum);
Return value:
Success: 0 failure: - 1
4.4 pause
Waiting signal
#include <unistd.h> //Wait for the signal to arrive (this function will block). Suspend the calling process until the signal is caught. This function is usually used to judge whether the signal has arrived. int pause(void);
It does not return - 1 until the signal is captured, and errno is set to EINTR.
4.5 signal
Processing method after the process receives the signal
1) Perform system default actions
For most signals, the default action of the system is to terminate the process.
2) Ignore this signal
There is no action after receiving this signal.
3) Execute custom signal processing functions
The signal is processed with a user-defined signal processing function.
Note: SIGKILL and SIGSTOP cannot change how signals are processed because they provide users with a reliable way to terminate processes.
Generate A signal that we can let execute custom signal processing functions. If there are functions A, B and C, how can we determine that only function A is called after the signal is generated, not function B or C. At this time, we need A rule that function A is called after the signal is generated. Just like traffic rules, red lights go green, and the signal registration function signal() does this.
#include <signal.h> typedef void (*sighandler_t)(int)；// Declaration of callback function sighandler_t signal(int signum,sighandler_t handler);
4.6 common signals
4.7 alarm
#include <unistd.h> //Send a SIGALRM signal to the calling process after second. The default action is to terminate the process calling the alarm function unsigned int alarm(unsigned int seconds);
Return value: if the timer has not been set before, or the timer has timed out, return 0, otherwise the remaining seconds of the timer will be returned and the timer will be reset
If the first alarm is not completed and the second alarm is executed, the time that the first alarm is not executed will be cleared directly
4.8 raise
#include <signal.h> //Send a signal to the calling process itself int raise(int sig);
In a single-threaded program it is equivalent to kill(getpid(), sig);
In a multithreaded program it is equivalent to pthread_kill(pthread_self(), sig);
4.9 abort
#include <stdlib.h> //By default, a SIGABRT signal is sent to the process, and the process will exit by default //Even if the SIGABRT signal is added to the blocking set, once the process calls the abort function, the process will still be terminated, and the buffer will be refreshed and the file descriptor will be closed when it is aborted void abort(void);
4.10 reentrant function
https://blog.csdn.net/lianghe_work/article/details/47611961
4.11 signal set
In order to facilitate the processing of multiple signals, a user process often needs to process multiple signals. The signal set (signal set) is introduced into the Linux system. This signal set is a bit similar to our QQ group. One signal is equivalent to one friend in the QQ group.
A signal set is a data type (sigset_t) used to represent multiple signals
The operations related to signal set mainly include the following functions:
#include <signal.h> //Initialize an empty signal set int sigemptyset(sigset_t *set); //Initialize a full signal set int sigfillset(sigset_t *set); //Query whether there is a signal in a signal set int sigismember(const sigset_t *set, int signum); //Add signal to signal set int sigaddset(sigset_t *set, int signum); //Remove signal from signal set int sigdelset(sigset_t *set, int signum);
4.12 signal blocking set
4.12.1 general
Each process has a blocking set. When a child process is created, the child process will inherit the blocking set of the parent process. The signal blocking set is used to describe which signals are blocked when delivered to the process (remember it when the signal occurs and notify the process of the signal until the process is ready).
**The so-called blocking is not to prohibit the transmission of signals, but to suspend the transmission of signals** If the blocked signal is deleted from the signal blocking set and the corresponding signal occurs when it is blocked, the process will receive the corresponding signal.
4.12.2 sigprocmask
#include <signal.h> //how: there are three ways to modify the signal blocking set: SIG_BLOCK: Add to signal blocking set set Signal set, the new signal mask is set Union of and old signal masks. SIG_UNBLOCK: Remove from signal blocking set set Signal set, removed from the current signal mask set Signal in. SIG_SETMASK: Set the signal blocking set to set The signal set is equivalent to clearing the contents of the original signal blocking set, and then set Resets the signal blocking set. //Set: address of the signal set to be operated. if set by NULL，The function only saves the current signal blocking set to oldset Yes. //oldset: save the address of the original signal blocking set //Check or modify the signal blocking set, and modify the blocking set of the process according to the method specified by how. The new signal blocking set is specified by set, while the original signal blocking set is saved by oldset. int sigprocmask(int how, const sigset_t *set, sigset_t *oldset);
Note: * * SIGKILL, SIGSTOP and other signals cannot be blocked, but sigprocmask() does not return errors when the set parameter contains these signals, but ignores them. In addition, blocking signals such as SIGFPE may lead to irreparable results, * * because these signals are generated by program errors, ignoring them can only lead to the failure of program execution and termination.
4.13 operation of reliable signal
Linux provides a more powerful sigaction() function, which can be used to check and change signal processing operations, support reliable and real-time signal processing, and support signal transmission information.
https://blog.csdn.net/lianghe_work/article/details/46804469
5. Interprocess communication (pipeline)
5.1 unknown pipeline
Pipe is also called anonymous pipe. It is the oldest form of IPC (interprocess communication) in UNIX system. All UNIX systems support this communication mechanism.
1. Half duplex, data can only flow in one direction at the same time.
2. Data can only be written from one end of the pipe and read from the other end.
3. Data written to the pipeline follows the first in, first out rule.
4. The data transmitted by the pipeline is unformatted, which requires that the reader and writer of the pipeline must agree on the format of the data in advance, such as how many bytes count as a message.
5. The pipeline is not an ordinary file and does not belong to a file system. It only exists in memory.
6. The pipeline corresponds to a buffer in memory. Different systems have different sizes.
7. Reading data from the pipeline is a one-time operation. Once the data is read, it will be discarded from the pipeline to free up space for writing more data.
8. The pipeline has no name and can only be used between processes with common ancestors (parent process and child process, or two brother processes, which are related).
We can compare it to a pipe in real life. One end of the pipe is stuffed with something, and the other end of the pipe takes something
Anonymous pipeline is a special type of file, which is embodied as two open file descriptors in the application layer.
5.2 pipe()
#include <unistd.h> //Create anonymous pipe //Filedes: the first address of the int array, which stores the file descriptors of the pipeline, filedes[0], filedes[1]. int pipe(int filedes[2]);
When a pipeline is established, it creates two file descriptors fd[0] and fd[1]. Where fd[0] is fixed for the read pipe and fd[1] is fixed for the write pipe. General file I/O functions can be used to operate on pipes (except lseek()).
5.3 characteristics of pipeline
Each pipeline has only one page as a buffer, which is used as a ring buffer. This access method is a typical "producer consumer" model. When the "producer" process has a lot of data to write, and whenever a page is full, it needs to sleep and wait for the "consumer" to read some data from the pipeline to make room for it. Accordingly, if there is no readable data in the pipeline, the "consumer" process will sleep and wait. The specific process is shown in the following figure:
By default, the main feature of reading and writing data from the pipeline is the blocking problem (this feature should be remembered). When there is no data in the pipeline, another process reads data from the pipeline with the read() function by default, which is blocked.
5.3.1 blocking 1
You can see that after the child process exits, the parent process doesn't need to read the data and is blocked in read
5.3.2 fcntl
How to solve it? You can set the blocking property of the file through the fcntl() function.
Set to block: fcntl(fd, F_SETFL, 0); Set to non blocking: fcntl(fd, F_SETFL, O_NONBLOCK);
5.3.3 blocking condition 2
Call the write() function to write data to the pipeline. When the buffer is full, write() will also block.
Blocked at 64
5.3.3 SIGPIPE
In the process of communication, after other processes finish first, when the current process writes data to the pipeline after the reading port of the current process is closed, the process where write() is located will exit (receiving SIGPIPE signal). The default action after receiving SIGPIPE is to interrupt the current process
The linux terminal pops up directly. It can be seen that the program is interrupted directly and there is no need to block it
5.4 named pipes
The difference between named pipeline (FIFO) and nameless pipeline is that it provides a path name associated with it and exists in the file system in the form of FIFO file. In this way, even processes that are not related to the creation process of FIFO can communicate with each other through FIFO as long as they can access the path. Therefore, processes that are not related through FIFO can also exchange
The difference from the anonymous pipeline is
1,FIFO Exists as a special file in the file system, but FIFO The contents of are stored in memory. 2,When used FIFO After your process exits, FIFO The file will continue to be saved in the file system for later use. 3,FIFO With names, unrelated processes can communicate by opening named pipes.
5.5 mkfifo
5.5.1 basic concepts
#include <sys/types.h> #include <sys/stat.h> //Pathname: common pathname, that is, the name of the created FIFO. //Mode: the permission of the file is the same as the mode parameter in the open() function of opening an ordinary file. int mkfifo( const char *pathname, mode_t mode);
Default actions for named pipes
For later operations, the named pipe is operated as an ordinary file: open(), write(), read(), close(). However, like nameless pipes, the blocking property of named pipes must be considered by default.
The following verification is the default feature, that is, the non blocking flag (O_NONBLOCK) is not specified when opening ().
When open() opens the FIFO in read-only mode, it is necessary to block a process from opening the FIFO for writing
When open() opens the FIFO in write only mode, it is necessary to block a process to open the FIFO for reading.
In a word * *, read-only waits for write only, write only waits for read-only, and only when both are executed, will it be executed next**
5.5.2 blocking 1
At the beginning of reading, the writer is not enabled and waits for blocking
After the writer is enabled, the writer detects the reader without blocking, and the reader also detects the writer without blocking
If you don't want to block when you open(), we can open the FIFO file in a readable and writable way, so that the open() function won't block.
5.5.3 blocking condition 2
If there is no data in FIFO, read() will also block when calling the read() function to read data from FIFO. This feature is the same as the nameless pipe
After the completion of the display, but before 5s, the reader side is blocked
5.5.4 blocking 3
In the communication process, if the writing process exits first, even if there is no data in the named pipeline, it will not be blocked when calling the read() function to read data from FIFO; If the write process runs again, the read () function is called to retrieve the FIF
When reading data in O, it resumes blocking.
5) During communication, after the read process exits, when the write process writes data to the named pipe, the write process will also exit (receiving SIGPIPE signal).
6) Call the write() function to write data to the FIFO. When the buffer is full, write() will also block.
5) And 6) these two features are the same as that of the unknown pipeline, which will not be verified here
Named pipes can be opened with a non blocking flag (O_NONBLOCK):
fd = open("my_fifo", O_WRONLY|O_NONBLOCK);
fd = open("my_fifo", O_RDONLY|O_NONBLOCK);
The named pipe opened by the non blocking flag (O_NONBLOCK) has the following characteristics:
1. Open it in read-only mode first. If no process has opened a FIFO for writing, the read-only open() succeeds and the open() is not blocked.
2. First, open it in write only mode. If no process has opened a FIFO for reading, write only open() will return -1. 3. read() and write() are not blocked when reading data in the named pipeline.
6. Interprocess communication (message queue)
6.1 general
Message queuing provides a simple and efficient way to transfer data between two unrelated processes. Its characteristics are as follows:
1) Message queue can realize random query of messages. Messages do not have to be read in first in first out order. They can be read according to the type of message during programming.
2) Message queuing allows one or more processes to write or read messages to it.
3) Like nameless pipes and named pipes, when a message is read from the message queue, the corresponding data in the message queue will be deleted.
4) Each message queue has a message queue identifier, which is unique in the whole system.
5) Message queue is a linked list of messages, which is stored in memory and maintained by the kernel** The message queue will be deleted only when the kernel restarts or manually deletes the message queue** If the message queue is not deleted manually, the message queue will always exist in the system.
#include <sys/msg.h> #include <sys/types.h> #include <sys/ipc.h> key_t ftok(const char *pathname, int proj_id); int msgget(key_t key, int msgflg); int msgrcv(int msqid, void *msg_ptr, size_t msg_sz, long int msgtype, int msgflg); int msgsnd(int msqid, const void *msg_ptr, size_t msg_sz, int msgflg); int msgctl(int msqid, int cmd, struct msqid_ds *buf);
In the message queue operation, the key value is equivalent to the address, the message queue identifier is equivalent to a specific bank, and the message type is equivalent to the safe number.
The same key value can guarantee the same message queue
The same message queue identifier ensures that different processes can communicate with each other
The same message type can ensure that a process fetches the other party's information.
key value
The interprocess communication mechanism provided by System V requires a key value. A unique message queue identifier can be obtained in the system through the key value. The key value can be manually specified or obtained through the ftok() function.
6.2 ftok
#include <sys/types.h> #include <sys/ipc.h> //Get key //Pathname: pathname //proj_id: item ID, non-zero integer (only the lower 8 bits are valid) key_t ftok(const char *pathname, int proj_id);
6.3 msgget
Create message queue
#include <sys/msg.h> //Create a new or open an existing message queue. When different processes call this function, the identifier of the same message queue can be obtained as long as the same key value is used //Key: the key value returned by ftok() //msgflg: identifies the behavior of the function and the permission of the message queue. The values are as follows: // IPC_CREAT: create a message queue. // IPC_EXCL: check whether the message queue exists. int msgget(key_t key, int msgflg);
ipcs -q query message queue
ipcrm -q [key] delete message queue
typedef struct _msg
{
long mtype; // Message type
char mtext[100]; // Message body
//... / / the body of a message can have multiple members
}MSG;
Below the type is the message body, which can have multiple members (body members can be of any data type)
6.4 msgsnd
Add a new message to the message queue.
include <sys/msg.h> //msqid: identifier of the message queue. //msgp: address of the message structure to be sent. //msgsz: the number of bytes in the message body. //msgflg: the control attribute of the function. Its values are as follows: //The 0: msgsnd() call blocks until the condition is met. //IPC_NOWAIT: if the message is not sent immediately, the process calling this function will return immediately. int msgsnd( int msqid, const void *msgp, size_t msgsz, int msgflg);
6.5 msgrcv
Receive a message from the message queue with the identifier msqid. Once the message is received successfully, the message is deleted in the message queue.
#include <sys/msg.h> //msqid: identifier of the message queue, representing which message column to get the message from. //msgp: the address where the message structure is stored. //msgsz: the number of bytes in the message body. //msgtyp: type of message. There are several types: msgtyp = 0: Returns the first message in the queue. msgtyp > 0: The message type in the return queue is msgtyp Message (common). msgtyp < 0: The value of message type in the return queue is less than or equal to msgtyp Absolute value message. If there are several such messages, take the message with the smallest type value. //msgflg: control properties of the function. The values are as follows: 0: msgrcv() Call blocking until the message is successfully received. MSG_NOERROR: If the number of returned message bytes is greater than nbytes Many bytes,The message will be truncated to nbytes Bytes without notifying the message sending process. IPC_NOWAIT: The calling process returns immediately. If no message is received, return immediately -1. ssize_t msgrcv( int msqid, void *msgp, size_t msgsz, long msgtyp, int msgflg );
Note: when getting a certain type of message, if there are multiple messages of this type in the queue, get the first added message, that is, the first in first out principle.
Success: read the length of the message
Failed: - 1
6.6 msgctl
Perform various controls on the message queue, such as modifying the properties of the message queue or deleting the message queue.
#include <sys/msg.h> //msqid: identifier of the message queue. //cmd: function control. The values are as follows: IPC_RMID: Delete by msqid Indicates the message queue, deletes it from the system and destroys the relevant data structure. IPC_STAT: take msqid The current value of each element in the relevant data structure is stored in the buf In the structure pointed to. Instead, back up the properties of the message queue to buf IPC_SET: take msqid The elements in the relevant data structure are set to buf The corresponding value in the structure pointed to. Equivalent to clearing the original attribute value of the message queue, and then buf To replace. //buf: msqid_ The address of DS data type, which is used to store or change the properties of message queue. int msgctl(int msqid, int cmd, struct msqid_ds *buf);
6.7 examples
Write end
Read end
7. Interprocess communication (shared memory)
7.1 general
Shared memory Is one of the simplest ways of inter process communication.
Shared memory allows two or more processes to access the same block of memory, just as the malloc() function returns pointers to the same physical memory area to different processes. When one process changes the content of this address, other processes will notice the change.
Features of shared memory:
Shared memory is one of the fastest ways to share data between processes. A process writes data to the shared memory area, and all processes sharing the memory area can immediately see the contents.
When using shared memory, we should pay attention to the mutual exclusion of access to a given storage area between multiple processes. If a process is writing data to the shared memory area, other processes should not read or write the data before it completes this step.
7.2 shmget
Create or open a shared memory area.
#include <sys/ipc.h> #include <sys/shm.h> key: Interprocess communication key value, ftok() Return value of. //size: the length of the shared storage segment (bytes). //shmflg: identifies the behavior of the function and the permission of shared memory. Its values are as follows: IPC_CREAT: If it doesn't exist, create it IPC_EXCL: If it already exists, it returns failure Bit or permission bit: after the shared memory bit or permission bit, you can set the access permission, format and password of the shared memory open() Functional mode_t Same( open() (please click this link), but the executable permission is not used. int shmget(key_t key, size_t size,int shmflg);
7.3 shmat
Map a shared memory segment to the data segment of the calling process. To understand it simply, let the process establish a relationship with the shared memory, and let a pointer of the process point to the shared memory.
#include <sys/types.h> #include <sys/shm.h> //shmid: shared memory identifier, return value of shmget(). //shmaddr: shared memory mapping address (if it is NULL, it will be automatically specified by the system). NULL is recommended. //shmflg: access permission and mapping condition of shared memory segment (usually 0). The specific values are as follows: 0: Shared memory has read and write permissions. SHM_RDONLY: Read only. SHM_RND: (shmaddr (valid only when not empty) void *shmat(int shmid, const void *shmaddr, int shmflg);
Return value:
Success: shared memory segment mapping address (equivalent to this pointer pointing to this shared memory)
Failed: - 1
7.4 shmdt
Separate the shared memory from the current process (just disconnect and do not delete the shared memory, which is equivalent to making the previous pointer to the shared memory no longer point to).
#include <sys/types.h> #include <sys/shm.h> //shmaddr: shared memory mapping address int shmdt(const void *shmaddr);
7.5 shmctl
Control of shared memory properties.
#include <sys/ipc.h> #include <sys/shm.h> //shmid: shared memory identifier. //cmd: control of function function, and its value is as follows: IPC_RMID: Delete.(Commonly used ) IPC_SET: set up shmid_ds Parameter, which is equivalent to replacing the original attribute value of shared memory with buf Property value in. IPC_STAT: preservation shmid_ds Parameter to back up the original attribute value of shared memory to buf Inside. SHM_LOCK: Lock shared memory segment( Super user ). SHM_UNLOCK: Unlock the shared memory segment. SHM_LOCK Used to lock memory and prohibit memory exchange. It does not mean that other processes are prohibited from accessing the shared memory after it is locked. Its real meaning is that locked memory is not allowed to be exchanged into virtual memory. The advantage of this is to keep the shared memory in memory all the time, so as to improve the performance of the program. //buf: shmid_ The address of DS data type, which is used to store or modify the properties of shared memory. int shmctl(int shmid, int cmd, struct shmid_ds *buf);
7.6 examples
Write end
Read end
8. Multithreading for multitasking
8.1 processes and threads
The thread is not only the execution of the application, but also the execution of the application.
In order for a process to complete certain work, the process must contain at least one thread.
Process: intuitively speaking, after the program stored on the hard disk runs, an independent memory body will be formed in the memory space. This memory body has its own address space and its own heap. The superior affiliated unit is the operating system. The operating system will allocate system resources in the unit of process, so we also say that process is the smallest unit of resource allocation.
Thread exists in the process and is the smallest unit of the operating system scheduling execution..
A process is a program with certain independent functions. It is a running activity on a data set. A process is an independent unit for resource allocation and scheduling. Thread is an entity of a process. It is the basic unit of CPU scheduling and dispatching. It is a smaller basic unit that can run independently than a process. The thread itself basically does not own system resources, but only some resources that are essential in operation (such as program counters, a set of registers and stacks), but it can share all the resources owned by the process with other threads belonging to the same process
The thread has its own stack, which still uses the address space of the process, but this space is marked as a stack by the thread. Each thread will have its own private stack, which can not be accessed by other threads.
The process maintains the resources (static resources) contained in the program, such as address space, open file handle set, file system status, signal processing handler, etc;
The running related resources (dynamic resources) maintained by the thread, such as running stack, scheduling related control information, signal set to be processed, etc;
Threads and processes have their own advantages and disadvantages in use: thread execution overhead is small, but it is not conducive to resource management and protection; The process is the opposite.
- All threads in the process share the same address space.
- Any variable or heap variable declared as static/extern can be read and written by all threads in the process.
- The only private storage that a thread really owns is the processor register.
- The thread stack can be shared with other threads by exposing the stack address.
8.2 pthread
Get thread number
#include <pthread.h> pthread_t pthread_self(void);
8.3 pthread_equal
Comparison of thread numbers
#include <pthread.h> int pthread_equal(pthread_t t1, pthread_t t2);
Return value:
Equal: not 0
Unequal: 0
8.4 pthread_create
Create a thread.
#include <pthread.h> thread: Thread identifier address. attr: Thread attribute structure address, usually set to NULL. start_routine: The entry address of the thread function. arg: Parameters passed to the thread function. int pthread_create( pthread_t *thread, const pthread_attr_t *attr, void *(*start_routine)(void *), void *arg );
Return value:
Success: 0 failure: not 0
pthread_ The thread created by create() runs from the specified callback function. After the function runs, the thread exits. Threads depend on the existence of the process and share the resources of the process. If the process that created the thread ends, the thread ends.
No parameter transfer (resource sharing)
Chuan Shen
8.5 pthread_join
Reclaim thread resources
Wait for the end of the thread (this function will block), and recycle the thread resources, similar to the wait() function of the process. If the thread has ended, the function returns immediately.
#include <pthread.h> thread: The number of the thread being waited for. retval: The address of the pointer used to store the exit state of the thread. int pthread_join(pthread_t thread, void **retval);
After you create a thread, you should recycle its resources, but use pthread_ The join () function blocks the caller, and Linux also provides the non blocking function pthread_detach() to reclaim the resources of the thread.
8.6 pthread_detach
Separating the calling thread from the current process does not mean that the thread does not depend on the current process. The purpose of thread separation is to hand over the recovery of thread resources to the system automatically, that is, when the separated thread ends, the system will automatically recover its resources. Therefore, this function will not block.
#include <pthread.h> thread: Thread number. int pthread_detach(pthread_t thread);
Return value:
Success: 0 failure: not 0
Notice that pthread is called_ Call pthread after detach()_ join() , pthread_ Join() will return immediately and the call fails.
8.7 pthread_exit
Exit the calling thread. Multiple threads in a process share the data segments of the process. Therefore, the resources occupied by the thread after exiting will not be released.
#include <pthread.h> retval: Stores a pointer to the exit state of the thread. void pthread_exit(void *retval);
8.8 thread private
In multithreaded programs, global variables are often used to realize data sharing among multiple functions. Since the data space is shared, global variables are also shared by all threads.
It can be seen that the modification of global variables by one thread will affect the access of the other thread.
However, sometimes it is necessary to provide thread private global variables in application design. This variable is only valid in threads, but it can be accessed across multiple functions. For example, in the program, each thread may need to maintain a linked list and use the same function to operate the linked list. The simplest way is to use thread related data structures with the same name but different variable addresses. Such a data structure can be maintained by Posix thread library and become thread specific data (or TSD).
8.8.1 pthread_key_create
Create a pthread of type_ key_ Private data variable of type T (key).
#include <pthread.h>
key: In distribution( malloc )Before thread private data, you need to create a key associated with thread private data( key )，The function of this key is to obtain access to the thread's private data.
destructor Clean up function address(For example: fun ). When the thread exits, if the thread private data address is not non NULL，This function is called automatically. The function pointer can be set to NULL ，In this way, the system will call the default cleanup function.
The callback function is defined as follows:
void fun(void *arg)
{
// arg is the key value
}
int pthread_key_create(pthread_key_t *key, void (*destructor)(void*));
8.8.2 pthread_key_delete
Unregister thread private data
int pthread_key_delete(pthread_key_t key);
8.8.3 pthread_setspecific
Set the association between thread private data (key) and value. Note that the value of value (not the content) is associated with key.
key: Thread private data. value: and key The associated pointer. int pthread_setspecific(pthread_key_t key, const void *value);
8.8.4 pthread_getspecific
Read the value associated with the thread's private data
void *pthread_getspecific(pthread_key_t key);
Return value:
Success: the value associated with thread private data (key) failed: NULL
// this is the test code for pthread_key
#include <stdio.h>
#include <pthread.h>
#include <unistd.h>
pthread_key_t key; // Private data, global variables
//Cleanup function
void echomsg(void* t)
{
printf("[destructor] thread_id = %lu, param = %p\n", pthread_self(), t);
}
void* child1(void* arg)
{
int i = 10;
pthread_t tid = pthread_self(); //Thread number
printf("\nset key value %d in thread %lu\n", i, tid);
pthread_setspecific(key, &i); // Set private data
printf("thread one sleep 2 until thread two finish\n\n");
sleep(2);
printf("\nthread %lu returns %d, add is %p\n",
tid, *((int*)pthread_getspecific(key)), pthread_getspecific(key));
}
void* child2(void* arg)
{
int temp = 20;
pthread_t tid = pthread_self(); //Thread number
printf("\nset key value %d in thread %lu\n", temp, tid);
pthread_setspecific(key, &temp); //Set private data
sleep(1);
printf("thread %lu returns %d, add is %p\n",
tid, *((int*)pthread_getspecific(key)), pthread_getspecific(key));
}
int main(void)
{
pthread_t tid1, tid2;
pthread_key_create(&key, echomsg); // establish
pthread_create(&tid1, NULL, child1, NULL);
pthread_create(&tid2, NULL, child2, NULL);
pthread_join(tid1, NULL);
pthread_join(tid2, NULL);
pthread_key_delete(key); // cancellation
return 0;
}
From the running results, each thread has no influence on its own private data operation. In other words, although the key has the same name and is global, the memory space accessed is not the same.
8.9 thread pool
8.9.1 basic principle of wire path pool
In the traditional server structure, there is often a general monitoring thread to monitor whether a new user connects to the server. Whenever a new user enters, the server opens a new thread to process the user's data packet. This thread only serves this user. When the user closes the connection with the server, the server destroys this thread.
However, the frequent development and destruction of threads greatly occupy the resources of the system, and in the case of a large number of users, the system will waste a lot of time and resources in order to develop and destroy threads. Thread pool It provides a solution to the contradiction between a large number of external users and the limited resources of the server.
Thread pool Different from the traditional processing method of a user corresponding to a thread, its basic idea is to open up some threads in memory at the beginning of the program. The number of threads is fixed. They form a class alone to shield external operations, and the server only needs to hand over the data packet to the thread pool.
When a new customer request arrives, instead of creating a new thread to serve it, select an idle thread from the "pool" to serve the new customer request. After the service is completed, the thread enters the idle thread pool.
If no thread is idle, the data packet will be temporarily accumulated and processed after the thread in the thread pool is idle. By reusing existing thread objects for multiple tasks, the overhead of creating and destroying Thread objects is reduced. When the client requests, the thread object already exists, which can improve the response time of the request, so as to improve the performance of the system service as a whole.
8.9.2 thread pool application example
Generally speaking, implementing a thread pool mainly includes the following components:
1) Thread manager: used to create and manage thread pools.
2) Worker thread: the thread that actually executes the task in the thread pool. When initializing threads, a fixed number of threads will be created in advance. In the pool, these initialized threads are generally idle, generally do not occupy CPU and occupy small memory space.
3) Task interface: the interface that must be implemented by each task. When there are executable tasks in the task queue of the thread pool, it is called to execute by idle working threads (the idle and busy of threads are realized through mutual exclusion). The task is abstracted to form an interface, which can make the thread pool independent of specific tasks.
4) Task queue: it is used to store unprocessed tasks and provide a buffer mechanism. There are several methods to realize this structure. The commonly used is queue, which mainly uses the first in first out principle. The other is a data structure such as linked list, which can dynamically allocate memory space for it. It is more flexible in application. This tutorial is the linked list used.
**When do I need to create a thread pool** In short, if an application needs to create and destroy threads frequently and the task execution time is very short, the overhead of thread creation and destruction can not be ignored. This is also an opportunity for the thread pool to appear. If the task execution time is negligible compared with the thread creation and destruction time, there is no need to use the thread pool.
8.9.3 thread pool example code
#ifndef __THREAD_POOL_H__
#define __THREAD_POOL_H__
#include <pthread.h>
/*********************************************************************
* The task callback function can also be modified as needed
*********************************************************************/
typedef void *(*pool_task_f)(void *arg);
/*********************************************************************
* task handle
*********************************************************************/
typedef struct _task{
pool_task_f process;/*Callback function, which will be called when the task is running. Note that it can also be declared in other forms*/
void *arg; /*Parameters of callback function*/
struct _task *next;
}pool_task;
/*********************************************************************
* Thread pool handle
*********************************************************************/
typedef struct
{
pthread_t *threadid; /* Thread number */
int threads_limit; /* The number of active threads allowed in the thread pool */
int destroy_flag; /* Do you want to destroy the thread pool, 0 destroy, 1 do not destroy*/
pool_task *queue_head; /* Linked list structure, all waiting tasks in the thread pool */
int task_in_queue; /* Number of tasks currently waiting in queue */
pthread_mutex_t queue_lock; /* lock */
pthread_cond_t queue_ready; /* Conditional variable */
}pool_t;
/*********************************************************************
*Function: Initialize the thread pool structure and create threads
*Parameters:
pool: Thread pool handle
threads_limit: Number of threads in the thread pool
*Return value: nothing
*********************************************************************/
void pool_init(pool_t *pool, int threads_limit);
/*********************************************************************
*Functions: Destroy the thread pool, and the tasks in the waiting queue will no longer be executed,
However, the running thread will continue to run, and then exit after running the task
*Parameters: Thread pool handle
*Return value: Success: 0, failure not 0
*********************************************************************/
int pool_uninit(pool_t *pool);
/*********************************************************************
*Function: Add a task to the thread pool
*Parameters:
pool: Thread pool handle
process: Task processing function
arg: Task parameters
*Return value: 0
*********************************************************************/
int pool_add_task(pool_t *pool, pool_task_f process, void *arg);
#endif
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <assert.h>
#include "thread_pool.h"
static void *pool_thread_server(void *arg);
/*********************************************************************
*Function: Initialize the thread pool structure and create threads
*Parameters:
pool: Thread pool handle
threads_limit: Number of threads in the thread pool
*Return value: nothing
*********************************************************************/
void pool_init(pool_t *pool, int threads_limit)
{
pool->threads_limit = threads_limit;
pool->queue_head = NULL;
pool->task_in_queue = 0;
pool->destroy_flag = 0;
/*Create space for thread ID*/
pool->threadid = (pthread_t *)calloc(threads_limit, sizeof(pthread_t));
int i = 0;
/*Initialize mutexes and condition variables*/
pthread_mutex_init(&(pool->queue_lock), NULL);
pthread_cond_init(&(pool->queue_ready), NULL);
/*Loop creation threads_limit threads*/
for (i = 0; i < threads_limit; i++){
pthread_create(&(pool->threadid[i]), NULL, pool_thread_server, pool);
}
return;
}
/*********************************************************************
*Function: Destroy the thread pool, and the tasks in the waiting queue will no longer be executed,
However, the running thread will continue to run, and then exit after running the task
*Parameters: Thread pool handle
*Return value: Success: 0, failure not 0
*********************************************************************/
int pool_uninit(pool_t *pool)
{
pool_task *head = NULL;
int i;
pthread_mutex_lock(&(pool->queue_lock));
if(pool->destroy_flag)/* Prevent two calls */
return -1;
pool->destroy_flag = 1;
pthread_mutex_unlock(&(pool->queue_lock));
/* Wake up all waiting threads and the thread pool will be destroyed */
pthread_cond_broadcast(&(pool->queue_ready));
/* Block and wait for the thread to exit, otherwise it will become a zombie */
for (i = 0; i < pool->threads_limit; i++)
pthread_join(pool->threadid[i], NULL);
free(pool->threadid);
/* Destroy waiting queue */
pthread_mutex_lock(&(pool->queue_lock));
while(pool->queue_head != NULL){
head = pool->queue_head;
pool->queue_head = pool->queue_head->next;
free(head);
}
pthread_mutex_unlock(&(pool->queue_lock));
/*Don't forget to destroy conditional variables and mutexes*/
pthread_mutex_destroy(&(pool->queue_lock));
pthread_cond_destroy(&(pool->queue_ready));
return 0;
}
/*********************************************************************
*Function: Add a task to the task queue
*Parameters:
pool: Thread pool handle
process: Task processing function
arg: Task parameters
*Return value: nothing
*********************************************************************/
static void enqueue_task(pool_t *pool, pool_task_f process, void *arg)
{
pool_task *task = NULL;
pool_task *member = NULL;
pthread_mutex_lock(&(pool->queue_lock));
if(pool->task_in_queue >= pool->threads_limit){
printf("task_in_queue > threads_limit!\n");
pthread_mutex_unlock (&(pool->queue_lock));
return;
}
task = (pool_task *)calloc(1, sizeof(pool_task));
assert(task != NULL);
task->process = process;
task->arg = arg;
task->next = NULL;
pool->task_in_queue++;
member = pool->queue_head;
if(member != NULL){
while(member->next != NULL) /* Add the task to the last position of the task chain */
member = member->next;
member->next = task;
}else{
pool->queue_head = task; /* If it's the first task, point to the head */
}
printf("\ttasks %d\n", pool->task_in_queue);
/* There are tasks in the waiting queue. Wake up a waiting thread */
pthread_cond_signal (&(pool->queue_ready));
pthread_mutex_unlock (&(pool->queue_lock));
}
/*********************************************************************
*Function: Take a task from the task queue
*Parameters: Thread pool handle
*Return value: task handle
*********************************************************************/
static pool_task *dequeue_task(pool_t *pool)
{
pool_task *task = NULL;
pthread_mutex_lock(&(pool->queue_lock));
/* Judge whether the thread pool is to be destroyed */
if(pool->destroy_flag){
pthread_mutex_unlock(&(pool->queue_lock));
printf("thread 0x%lx will be destroyed\n", pthread_self());
pthread_exit(NULL);
}
/* If the wait queue is 0 and the thread pool is not destroyed, it is blocked */
if(pool->task_in_queue == 0){
while((pool->task_in_queue == 0) && (!pool->destroy_flag)){
printf("thread 0x%lx is waitting\n", pthread_self());
/* Note: pthread_cond_wait is an atomic operation. It will be unlocked before waiting and locked after waking up */
pthread_cond_wait(&(pool->queue_ready), &(pool->queue_lock));
}
}else{
/* Subtract 1 from the length of the waiting queue and take out the first element in the queue */
pool->task_in_queue--;
task = pool->queue_head;
pool->queue_head = task->next;
printf("thread 0x%lx received a task\n", pthread_self());
}
pthread_mutex_unlock(&(pool->queue_lock));
return task;
}
/*********************************************************************
*Function: Add a task to the thread pool
*Parameters:
pool: Thread pool handle
process: Task processing function
arg: Task parameters
*Return value: 0
*********************************************************************/
int pool_add_task(pool_t *pool, pool_task_f process, void *arg)
{
enqueue_task(pool, process, arg);
return 0;
}
/*********************************************************************
*Function: Thread pool server
*Parameters: slightly
*Return value: slightly
*********************************************************************/
static void *pool_thread_server(void *arg)
{
pool_t *pool = NULL;
pool = (pool_t *)arg;
while(1){
pool_task *task = NULL;
task = dequeue_task(pool);
/*Call the callback function to execute the task*/
if(task != NULL){
printf ("thread 0x%lx is busy\n", pthread_self());
task->process(task->arg);
free(task);
task = NULL;
}
}
/*This sentence should be unreachable*/
pthread_exit(NULL);
return NULL;
}
8.9.4 test code
#include <stdio.h>
#include <unistd.h>
#include "thread_pool.h"
void *task_test(void *arg)
{
printf("\t\tworking on task %d\n", (int)arg);
sleep(1); /*Take a second off to extend the execution time of the task*/
return NULL;
}
void thread_pool_demo(void)
{
pool_t pool;
int i = 0;
pool_init(&pool, 2);//Initialize a thread pool in which 2 threads are created
sleep(1);
for(i = 0; i < 5; i++){
sleep(1);
pool_add_task(&pool, task_test, (void *)i);//Add a task
}
sleep(4);
pool_uninit(&pool);//Delete thread pool
}
int main (int argc, char *argv[])
{
thread_pool_demo();
return 0;
}
9. Multi task synchronization and mutual exclusion
9.1 basic introduction
Modern operating systems are basically multitasking operating systems, that is, there are a large number of schedulable entities running at the same time. In a multitasking operating system, multiple tasks running simultaneously may:
-
All need to access / use the same resource
-
There are dependencies between multiple tasks, and the operation of a task depends on another task. These two situations are the most basic problems encountered in multitask programming and the core problems in multitask programming. Synchronization and mutual exclusion are used to solve these two problems.
Mutual exclusion: refers to several program fragments walking between different tasks. When a task runs one of the program fragments, other tasks cannot run any of them. They can only run after the task runs the program fragment. The most basic scenario is: a public resource can only be used by one process or thread at the same time, and multiple processes or threads cannot use public resources at the same time.
**Synchronization: * * refers to A number of program fragments between different tasks. Their operation must be in strict accordance with A specified order, which depends on the specific task to be completed. The most basic scenario is: * * two or more processes or threads coordinate synchronously during operation and run in A predetermined order** For example, the operation of task A depends on the data generated by task B.
Obviously, synchronization is a more complex mutual exclusion, and mutual exclusion is a special synchronization. In other words, mutual exclusion means that two tasks cannot run at the same time. They will repel each other. They must wait for one thread to run before the other can run, and synchronization cannot run at the same time, but they must run the corresponding threads in a certain order (also a kind of mutual exclusion)! Therefore, mutual exclusion has uniqueness and exclusivity, but mutual exclusion does not limit the running order of tasks, that is, tasks are out of order, while synchronous tasks have sequential relationship.
9.2 mutex
9.2.1 introduction
In a multitasking operating system, multiple tasks running at the same time may need to use the same resource.
This process is somewhat similar to that in the company department, while I use the printer to print things (not finished printing), others happen to use the printer to print things at the same time. If nothing is done, the printed things must be disordered.
There is such a lock in the thread - mutex. Mutex is a simple locking method to control access to shared resources. Mutex has only two states, lock and unlock.
The operation flow of mutex is as follows:
1) Lock the mutex before accessing the post critical area of shared resources.
2) Release the lock on the mutex guide after the access is complete.
3) After locking the mutex, any other thread attempting to lock the mutex again will be blocked until the lock is released.
9.2.2 pthread_mutex_init
Initialize a mutex.
mutex: Mutex address. Type is pthread_mutex_t .
attr: Set the attribute of mutex. Usually, you can use the default attribute to attr Set as NULL.
You can use macros PTHREAD_MUTEX_INITIALIZER Statically initialize the mutex, for example:
pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
This method is equivalent to using NULL designated attr Parameter call pthread_mutex_init () To complete dynamic initialization, the difference is PTHREAD_MUTEX_INITIALIZER Macros do not perform error checking.
int pthread_mutex_init(pthread_mutex_t *mutex, const pthread_mutexattr_t *attr);
Return value:
Success: 0. The successfully applied lock is open by default. Failure: non-0 error code
9.2.3 pthread_mutex_lock | pthread_mutex_trylock
Lock the mutex. If the mutex is locked, the caller will block it until the mutex is unlocked.
int pthread_mutex_lock(pthread_mutex_t *mutex);
When calling this function, if the mutex lock is not locked, it will be locked and return 0; If the mutex has been locked, the function directly returns failure, that is, EBUSY.
int pthread_mutex_trylock(pthread_mutex_t *mutex);
9.2.4 pthread_mutex_unlock
int pthread_mutex_unlock(pthread_mutex_t * mutex);
9.2.5 pthread_mutex_destory
int pthread_mutex_destroy(pthread_mutex_t *mutex);
9.2.6 examples
9.3 read write lock
9.3.1 introduction
When a thread already holds a mutex, the mutex blocks all threads trying to enter the critical zone. However, consider a case where the thread currently holding the mutex only wants to read and access the shared resource, while several other threads also want to read the shared resource. However, due to the exclusivity of the mutex, all other threads cannot obtain the lock, so they cannot read and access the shared resource. However, in fact, multiple threads reading and accessing the shared resource at the same time will not cause problems.
In the data reading and writing operations, there are more reading operations and less writing operations, such as the application of reading and writing database data. In order to meet the current requirement of allowing multiple reads but only one write, threads provide Read write lock To achieve.
Read write lock The features are as follows:
1) If there are other threads reading data, other threads are allowed to perform read operations, but write operations are not allowed.
2) If other threads write data, other threads are not allowed to read or write.
Read / write locks are divided into read locks and write locks. The rules are as follows:
1) If a thread applies for a read lock, other threads can apply for a read lock, but cannot apply for a write lock.
2) If a thread applies for a write lock, other threads cannot apply for a read lock or a write lock.
Parameters passed are basically consistent with the mutex lock
9.4 semaphore
9.4.1 general
Semaphore overview
Semaphores are widely used for synchronization and mutual exclusion between processes or threads. Semaphores are essentially a non negative integer counter, which is used to control access to public resources.
During programming, you can judge whether you have access to public resources according to the result of operating the semaphore value. When the semaphore value is greater than 0, you can access, otherwise it will be blocked. PV primitive is the operation of semaphore. A P operation reduces the semaphore by 1, and a V operation increases the semaphore by 1.
Semaphores are mainly used for synchronization and mutual exclusion between processes or threads.
For mutual exclusion
For synchronization
In POSIX standard, semaphores are divided into two types, one is unknown semaphore and the other is famous semaphore.
Anonymous semaphores are generally used for synchronization or mutual exclusion between threads
Well known semaphores are generally used for inter process synchronization or mutual exclusion. The difference between them is similar to that between pipes and named pipes. Nameless semaphores are stored directly in memory, while famous semaphores require the creation of a file.
9.4.2 unknown semaphore
Basic operation of unknown semaphore
sem_init
Create a semaphore and initialize its value. An unnamed semaphore must be initialized before being used
#include <semaphore.h> sem: Address of the semaphore. pshared: Equal to 0, the semaphore is shared between processes (commonly used); Not equal to 0, semaphores are shared between processes. value: The initial value of the semaphore. int sem_init(sem_t *sem, int pshared, unsigned int value);
sem_wait | sem_trywait
Semaphore P operation (minus 1)
Reduce the value of the semaphore by 1. Before operation, check whether the value of semaphore (sem) is 0. If the semaphore is 0, this function will block, and the minus 1 operation will not be carried out until the semaphore is greater than 0.
#include <semaphore.h>, sem: Address of the semaphore. int sem_wait(sem_t *sem);
Reduce the semaphore by 1 in a non blocking manner. If the value of the semaphore is equal to 0 before the operation, the operation on the semaphore fails and the function returns immediately.
int sem_trywait(sem_t *sem);
sem_post
Semaphore V operation (plus 1)
Increase the semaphore value by 1 and send a signal to wake up the waiting thread (sem_wait()).
#include <semaphore.h>, sem: Address of the semaphore. int sem_post(sem_t *sem);
sem_getvalue
Get the semaphore value identified by sem and save it in sval.
#include <semaphore.h> sem: Semaphore address. sval: Address where the semaphore value is saved. int sem_getvalue(sem_t *sem, int *sval);
sem_destroy
#include <semaphore.h> int sem_destroy(sem_t *sem);
example
9.4.3 famous semaphore
create
Use when a named semaphore exists: sem_t *sem_open(const char *name, int oflag); Use when the named semaphore does not exist: sem_t *sem_open(const char *name, int oflag, mode_t mode, unsigned int value);
Name: semaphore file name. Note that pathnames cannot be specified. Because of the famous semaphore, it is placed in / dev/shm by default
flags: sem_ The behavior flag of the open() function.
mode: setting of file permissions (readable, writable and executable).
Value: initial value of semaphore.
sem_close
Turn off the semaphore.
int sem_close(sem_t *sem);
sem_unlink
Delete the file named semaphore.
int sem_unlink(const char *name);
example
void printer(sem_t* sem, char* str)
{
sem_wait(sem); //Semaphore minus one
while (*str != '\0')
{
putchar(*str);
fflush(stdout);
str++;
sleep(1);
}
printf("\n");
sem_post(sem); //Semaphore plus one
}
int main(int argc, char* argv[])
{
pid_t pid;
sem_t* sem = NULL;
pid = fork(); //Create process
if (pid < 0) { //error
perror("fork error");
}
else if (pid == 0) { //Subprocess
//It is very similar to the opening method of open(). As long as different processes have the same name, they open the same named semaphore
sem = sem_open("name_sem1", O_CREAT | O_RDWR, 0666, 1); //The semaphore value is 1
if (sem == SEM_FAILED) { //Failed to create a named semaphore
perror("sem_open");
return -1;
}
char* str1 = "hello";
printer(sem, str1); //Print
sem_close(sem); //Turn off the named semaphore
_exit(1);
}
else if (pid > 0) { //Parent process
//It is very similar to the opening method of open(). As long as different processes have the same name, they open the same named semaphore
sem = sem_open("name_sem1", O_CREAT | O_RDWR, 0666, 1); //The signal value is 1
if (sem == SEM_FAILED) {//Failed to create a named semaphore
perror("sem_open");
return -1;
}
char* str2 = "world";
printer(sem, str2); //Print
sem_close(sem); //Turn off the named semaphore
waitpid(pid, NULL, 0); //Wait for the child process to end
}
sem_unlink("name_sem1");//Delete a named semaphore
return 0;
}
#include <semaphore.h>
sem: semaphore address.
sval: the address where the semaphore value is saved.
int sem_getvalue(sem_t *sem, int *sval);
#### sem_destroy ```c #include <semaphore.h> int sem_destroy(sem_t *sem);
example
[external chain picture transferring... (img-5z4gqOQs-1649984831176)]
9.4.3 famous semaphore
create
Use when a named semaphore exists: sem_t *sem_open(const char *name, int oflag); Use when the named semaphore does not exist: sem_t *sem_open(const char *name, int oflag, mode_t mode, unsigned int value);
Name: semaphore file name. Note that pathnames cannot be specified. Because of the famous semaphore, it is placed in / dev/shm by default
[external chain pictures are being transferred... (img-KoyI6jaw-1649984831177)]
flags: sem_ The behavior flag of the open() function.
mode: setting of file permissions (readable, writable and executable).
Value: initial value of semaphore.
sem_close
Turn off the semaphore.
int sem_close(sem_t *sem);
sem_unlink
Delete the file named semaphore.
int sem_unlink(const char *name);
example
void printer(sem_t* sem, char* str)
{
sem_wait(sem); //Semaphore minus one
while (*str != '\0')
{
putchar(*str);
fflush(stdout);
str++;
sleep(1);
}
printf("\n");
sem_post(sem); //Semaphore plus one
}
int main(int argc, char* argv[])
{
pid_t pid;
sem_t* sem = NULL;
pid = fork(); //Create process
if (pid < 0) { //error
perror("fork error");
}
else if (pid == 0) { //Subprocess
//It is very similar to the opening method of open(). As long as different processes have the same name, they open the same named semaphore
sem = sem_open("name_sem1", O_CREAT | O_RDWR, 0666, 1); //The semaphore value is 1
if (sem == SEM_FAILED) { //Failed to create a named semaphore
perror("sem_open");
return -1;
}
char* str1 = "hello";
printer(sem, str1); //Print
sem_close(sem); //Turn off the named semaphore
_exit(1);
}
else if (pid > 0) { //Parent process
//It is very similar to the opening method of open(). As long as different processes have the same name, they open the same named semaphore
sem = sem_open("name_sem1", O_CREAT | O_RDWR, 0666, 1); //The signal value is 1
if (sem == SEM_FAILED) {//Failed to create a named semaphore
perror("sem_open");
return -1;
}
char* str2 = "world";
printer(sem, str2); //Print
sem_close(sem); //Turn off the named semaphore
waitpid(pid, NULL, 0); //Wait for the child process to end
}
sem_unlink("name_sem1");//Delete a named semaphore
return 0;
}
[external chain picture transferring... (img-2HDaQ4Sc-1649984831178)]