1. grep tool
grep is a line filter tool; Used to filter rows based on keywords
Syntax:
#grep [options] 'keyword' file name
Common options:
OPTIONS:
-i: Case insensitive
-v: Finds a row that does not contain the specified content,Reverse selection
-w: Search by word
-o: Print matching keywords
-c: Count the number of rows matched
-n: set number
-r: Traverse directory lookup layer by layer
-A: Displays the matching line and how many lines follow
-B: Displays the matching line and how many lines precede it
-C: Show how many lines before and after the matching line
-l: Only matching file names are listed
-L: List mismatched file names
-e: Use regular matching
-E:Use extended regular matching
^key:Start with keyword
key$:End with keyword
^$:Match blank lines
--color=auto : You can add the color display to the found keyword part
Color display (alias settings):
Temporary settings: # alias grep='grep --color=auto '/ / only valid for the current terminal and user Permanent settings: 1)Global (effective for all users) vim /etc/bashrc alias grep='grep --color=auto' source /etc/bashrc 2)Local (for a specific user) vim ~/.bashrc alias grep='grep --color=auto' source ~/.bashrc
For example:
grep -i 'root' /etc/passwd //Ignore case matching rows containing root grep -w 'ftp' /etc/passwd //Exactly match ftp words grep -w 'hello' /etc/passwd //Accurately match Hello words; Add your own line containing hello to the file grep -wo 'ftp' /etc/passwd //Print the matching keyword ftp grep -n 'root' /etc/passwd //Print line numbers that match the root keyword grep -ni 'root' /etc/passwd //Ignore case matching statistics for rows containing the keyword root grep -nic 'root' /etc/passwd //Ignore case matching and count the number of rows containing the keyword root grep -i '^root' /etc/passwd //Ignore case matching lines starting with root grep 'bash$' /etc/passwd //Match lines ending in bash grep -n '^$' /etc/passwd //Match blank lines and print line numbers grep '^#' /etc/passwd //Match lines beginning with # sign grep -v '^#' /etc/passwd //Matches lines that do not begin with a # sign grep -A5 'mail' /etc/passwd //The match contains the mail keyword and its next 5 lines grep -B5 'mail' /etc/passwd //The match contains the mail keyword and its first 5 lines grep -C5 'mail' /etc/passwd //The match contains the mail keyword and its first and last 5 lines
2. cut tool
cut is a tool for intercepting columns
Syntax:
cut Option file name
Common options:
-c: Split in characters,intercept -d: Custom separator, tab by default\t -f: And -d Used together to specify which area to intercept
For example:
cut -d: -f1 1.txt //Divide with colon and intercept the contents of column 1 cut -d: -f1,6,7 1.txt //Divide with colon and intercept the contents of columns 1, 6 and 7 cut -c4 1.txt //Intercept the 4th character of each line in the file cut -c1-4 1.txt //Intercept 1-4 characters per line in the file cut -c4-10 1.txt //Intercept 4-10 characters per line in the file cut -c5- 1.txt //Intercept all the following characters from the fifth character
Exercise: list the operation levels of your current system with gadgets. The operation level is generally 5 or 3
- How to view system operation level
- Command runlevel
- File / etc/inittab
- How to filter run levels
runlevel | cut -c3 runlevel | cut -d ' ' -f2 grep -v '^#' /etc/inittab | cut -d: -f2 grep '^id' /etc/inittab | cut -d: -f2 grep "initdefault:$" /etc/inittab | cut -c4 grep -v ^# /etc/inittab | cut -c4 grep 'id:' /etc/inittab | cut -d: -f2 cut -d':' -f2 /etc/inittab | grep -v ^# cut -c4 /etc/inittab | tail -1 cut -d: -f2 /etc/inittab | tail -1
3. sort tool
sort tool is used for sorting; It takes each line of the file as a unit, compares the ASCII values from the first character back, and finally outputs them in ascending order.
Syntax and options
-u : Remove duplicate lines -r : Sort in descending order. The default is ascending order -o : Output the sorting results to a file,Similar redirection symbol > -n: Sort by number. By default, sort by character -t : Separator -k : The first N column -b : Ignore leading spaces. -R : Random sorting, the results of each run are different
Examples
head /etc/passwd > 1.txt sort -n -t: -k3 1.txt //Sort in ascending order according to the user's uid sort -nr -t: -k3 1.txt //Descending by user uid sort -nr -t: -k3 1.txt -o 2.txt sort -n 2.txt //Sort by number sort -nu 2.txt //Sort by number and de duplicate sort -nr 2.txt sort -nru 2.txt sort -nru 2.txt sort -n 2.txt -o 3.txt Sort by number and redirect the results to a file sort -R 2.txt sort -u 2.txt
4. uniq tools
uniq is used to remove consecutive duplicate lines
-i: ignore case -c: Count the number of repeated lines -d: Show only duplicate lines For example: uniq 2.txt uniq -d 2.txt uniq -dc 2.txt
5. tee tools
tee tool reads and writes from standard input to standard output and file, that is, bidirectional overlay redirection (screen output | text input)
Options: -a Bidirectional additional redirection echo hello world echo hello world | tee file1 cat file1 echo 999 | tee -a file1 cat file1 grep -v '^#' vsftpd.conf | grep -v '^$' | tee vsftpd.conf.back
6. diff tool
diff tool is used to compare different files line by line
Note: diff describes two files differently by telling us how to change the first file and then match the second file.
Syntax:
diff [option] File 1 file 2
Common options:
| option | meaning |
|---|---|
| -b | Do not check spaces |
| -B | Do not check blank lines |
| -i | Do not check case |
| -w | Ignore all spaces |
| –normal | Normal format display (default) |
| -c | Context format display |
| -u | Merge format display |
For example:
- Compare the similarities and differences between the two ordinary documents, and prepare the documents:
# cat file1 aaaa 111 hello world 222 333 bbb # cat file2 aaa hello 111 222 bbb 333 world
1) Normal display
diff Purpose: file1 How to change talents and file2 matching [root@MissHou ~]# diff file1 file2 1c1,2 The first line of the first file needs to be changed(c=change)To match lines 1 to 2 of the second file < aaaa Less than sign"<"Represents the file on the left(file1)Document content ------Represents the separator > aaa Greater than sign">"Represents the file on the right(file2)Document content > hello 3d3 Delete line 3 of the first file(d=delete)Before it can match the third line of the second file < hello world 5d4 Line 5 of the first file can match line 4 of the second file only after being deleted < 333 6a6,7 Add line 6 of the first file(a=add)The content can be matched with lines 6 to 7 of the second file > 333 The contents to be added in the second document are 333 and 333 world > world
2) Context format display
[root@MissHou ~]# diff -c file1 file2 The first two lines mainly list the file name and time stamp of the file to be compared; Symbol before file name***express file1，---express file2 *** file1 2019-04-16 16:26:05.748650262 +0800 --- file2 2019-04-16 16:26:30.470646030 +0800 ***************I'm a separator *** 1,6 **** with***Opening representation file1 File, 1,6 Indicates 1 to 6 lines ! aaaa! Indicates that the line needs to be modified to match the second file 111 - hello world -Indicates that the line needs to be deleted to match the second file 222 - 333 -Indicates that the line needs to be deleted to match the second file bbb --- 1,7 ---- with---Opening representation file2 File, 1,7 Indicates lines 1 to 7 ! aaa Indicates that the first file needs to be modified to match the second file ! hello Indicates that the first file needs to be modified to match the second file 111 222 bbb + 333 Indicates that the first file needs to add this line to match the second file + world Indicates that the first file needs to add this line to match the second file
3) Merge format display
[root@MissHou ~]# diff -u file1 file2 The first two lines mainly list the file name and time stamp of the file to be compared; Symbol before file name---express file1，+++express file2 --- file1 2019-04-16 16:26:05.748650262 +0800 +++ file2 2019-04-16 16:26:30.470646030 +0800 @@ -1,6 +1,7 @@ -aaaa +aaa +hello 111 -hello world 222 -333 bbb +333 +world
- Compare the differences between the two directories
By default, the contents of the same files in the two directories will also be compared [root@MissHou tmp]# diff dir1 dir2 diff dir1/file1 dir2/file1 0a1 > hello Only in dir1: file3 Only in dir2: test1 If you only need to compare the differences between the files in the two directories, you don't need to further compare the file contents, you need to add-q option [root@MissHou tmp]# diff -q dir1 dir2 Files dir1/file1 and dir2/file1 differ Only in dir1: file3 Only in dir2: test1
Other tips:
Sometimes we need to take one file as the standard to modify other files, and when there are many modifications, we can complete it by patching.
1)First find out the differences between files, and then output to a file [root@MissHou ~]# diff -uN file1 file2 > file.patch -u:Context mode -N:Treat non-existent files as empty files 2)Patch different contents to the file [root@MissHou ~]# patch file1 file.patch patching file file1 3)Test verification [root@MissHou ~]# diff file1 file2
7. paste tool
The paste tool is used to merge file lines
Common options: -d: Custom spacer, default is tab -s: Serial processing, non parallel
8. tr tools
tr is used for character conversion, replacement and deletion; It is mainly used to delete control characters or convert characters in files
Syntax:
Usage 1: submit the execution result of the command to tr Processing, where string1 For query, string2 For conversion processing # commands|tr 'string1' 'string2' Usage 2: tr The processing content comes from the file, remember to use"<"Standard input # tr 'string1' 'string2' < filename Usage 3: Match string1 Carry out corresponding operations, such as deletion # tr options 'string1' < filename
Common options:
-d Delete all input characters in string 1. -s Delete all repeated character sequences and keep only the first one; The recurring string is about to be compressed into one string
Constant matching string:
| character string | meaning |
|---|---|
| a-z or [: lower:] | Match all lowercase letters |
| A-Z or [: upper:] | Match all uppercase letters |
| 0-9 or [: digit:] | Match all numbers |
| [:alnum:] | Match all letters and numbers |
| [:alpha:] | Match all letters |
| [:blank:] | All horizontal blanks |
| [:punct:] | Match all punctuation marks |
| [:space:] | All horizontal or vertical spaces |
| [:cntrl:] | All control characters |
| \n | Ctrl-J wrap |
| \r | Ctrl-M enter |
| \t | Ctrl-I tab |
| \f | Ctrl-L line feed |
For example:
# cat 3.txt create the file yourself for testing ROOT:x:0:0:root:/root:/bin/bash bin:x:1:1:bin:/bin:/sbin/nologin daemon:x:2:2:daemon:/sbin:/sbin/nologin adm:x:3:4:adm:/var/adm:/sbin/nologin lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin sync:x:5:0:sync:/sbin:/bin/sync shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown halt:x:7:0:halt:/sbin:/sbin/halt mail:x:8:12:mail:/var/spool/mail:/sbin/nologin uucp:x:10:14:uucp:/var/spool/uucp:/sbin/nologin boss02:x:516:511::/home/boss02:/bin/bash vip:x:517:517::/home/vip:/bin/bash stu1:x:518:518::/home/stu1:/bin/bash mailnull:x:47:47::/var/spool/mqueue:/sbin/nologin smmsp:x:51:51::/var/spool/mqueue:/sbin/nologin aaaaaaaaaaaaaaaaaaaa bbbbbb111111122222222222233333333cccccccc hello world 888 666 777 999 tr -d '[:/]' < 3.txt Delete in file:and/ cat 3.txt |tr -d '[:/]'Delete in file:and/ tr '[0-9]' '@' < 3.txt Replace the number in the file with@Symbol tr '[a-z]' '[A-Z]' < 3.txt Replace lowercase letters with uppercase letters in the file tr -s '[a-z]' < 3.txt Match lowercase letters and compress duplicates into one tr -s '[a-z0-9]' < 3.txt Match lowercase letters and numbers and compress duplicates into one tr -d '[:digit:]' < 3.txt Delete numbers from file tr -d '[:blank:]' < 3.txt Delete horizontal blank tr -d '[:space:]' < 3.txt Delete all horizontal and vertical whitespace
Small trial ox knife
- Use gadgets to intercept the current host IP respectively; Intercept NETMASK; Intercepting broadcast address; Intercept MAC address
ifconfig eth0|grep 'Bcast'|tr -d '[a-zA-Z ]'|cut -d: -f2,3,4 10.1.1.1:10.1.1.255:255.255.255.0 ifconfig eth0|grep 'Bcast'|tr -d '[a-zA-Z ]'|cut -d: -f2,3,4|tr ':' '\n' 10.1.1.1 10.1.1.255 255.255.255.0 #ifconfig eth0|grep 'HWaddr'|cut -d: -f2-|cut -d' ' -f4 00:0C:29:25:AE:54 #ifconfig eth0|grep 'HW'|tr -s ' '|cut -d' ' -f5 00:0C:29:B4:9E:4E # ifconfig eth1|grep Bcast|cut -d: -f2|cut -d' ' -f1 # ifconfig eth1|grep Bcast|cut -d: -f2|tr -d '[ a-zA-Z]' # ifconfig eth1|grep Bcast|tr -d '[:a-zA-Z]'|tr ' ' '@'|tr -s '@'|tr '@' '\n'|grep -v ^$ # ifconfig eth0|grep 'Bcast'|tr -d [:alpha:]|tr '[ :]' '\n'|grep -v ^$ # ifconfig eth1|grep HWaddr|cut -d ' ' -f11 # ifconfig eth0|grep HWaddr|tr -s ' '|cut -d' ' -f5 # ifconfig eth1|grep HWaddr|tr -s ' '|cut -d' ' -f5 # ifconfig eth0|grep 'Bcast'|tr -d 'a-zA-Z:'|tr ' ' '\n'|grep -v '^$'
- Save the user name, password and default shell of all ordinary users in the system to a file. It is required to separate the user name, password and default shell with tab key
grep 'bash$' /etc/passwd |grep -v 'root'|cut -d: -f1,2,7|tr ':' '\t' |tee abc.txt