network programming
outline
The Linux operating system and its components
shell command
file operation command
IO
document
byte string (bytes)
file read and write
other operations
flush buffer
file offset
file descriptor
file management function
Basics of Network Programming
OSI seven layer model
Four-layer model (TCP/IP model)
data transfer process
Network protocol
Network Basic Concepts
transport layer service
Connection-oriented transmission service (data transmission based on TCP protocol)
Connectionless transmission service (data transmission based on UDP protocol)
socket socket programming
Introduction to sockets
tcp socket programming
Server process
client process
tcp socket data transmission characteristics
Network send and receive buffer
tcp sticky packet
UDP socket programming
Server process
client process
socket socket properties
struct module for data packaging
HTTP transport
HTTP protocol (Hypertext Transfer Protocol)
HTTP request (request)
http response (response)
The Linux operating system and its components
The role of the operating system
An operating system (OS) is a computer program that manages computer hardware and software resources, and is also the core and cornerstone of a computer system. The operating system needs to handle basic tasks such as managing and configuring memory, prioritizing the supply and demand of system resources, controlling input and output devices, operating the network, and managing the file system. The operating system also provides an interface for the user to interact with the system.
Composition of Linux operating system
A typical Linux operating system consists of: Linux kernel, file system, command line shell, graphical interface and desktop environment, and includes various tools and application software.
Linux kernel: the core code of the Linux operating system
File system: usually refers to the system that manages disk data, and can store data in the form of directories or files. Each file system has its own special format and functions
shell command: Receive user commands, then call the corresponding application program, and feed back the information specified by the user according to the instructions entered by the user.
shell command
file operation command
Directory structure under linux
IO
definition
The operation of data exchange in memory is considered as IO operation, such as interacting with the terminal, interacting with the disk, interacting with the network, etc.
Program classification
IO-intensive program: There are a lot of IO operations during program execution, but less cpu operations. It consumes less cpu and takes a long time.
Computation-intensive program: the program runs more calculations and relatively less IO operations. High cpu consumption, fast execution, almost no blocking.
document
A file is a piece of data stored on a persistent storage device (hard disk, U disk, CD...). From the perspective of format encoding, it is divided into text files (which will be automatically decoded into characters after opening), binary files (video, audio, etc.). Think of a file as a type of object in Python, similar to the other types you have learned before.
byte string (bytes)
The concept of byte strings was introduced in python3. Unlike str, byte strings express data in byte sequence values, which is more convenient for processing two-process data. Therefore, byte strings are a common way to represent binary data in python3.
Ordinary ascii encoded strings can be converted to byte strings by adding b in front, for example: b'hello'
String to byte string method: str.encode()
Byte string to string method: bytes.decode()
file read and write
The basic operation steps to read and write files are: open the file, read and write the file, close the file
1. Open the file
file_object = open(file_name, access_mode='r', buffering=-1)
Function: Open a file and return a file object.
parameter: file_name file name;
access_mode How to open the file,If not written, the default is'r'
file mode operate
r Open for reading The file must exist
w open for writing
If the file does not exist, it will be created, and if it exists, the original content will be cleared
a open in append mode
r+ Open in read-write mode The file must exist
w+ Open a file in read-write mode
If it does not exist, it will be created, if it exists, the original content will be cleared
a+ Open in read-write mode Append mode
rb Open in binary read mode with r
wb Open in binary write mode with w
ab Open in binary append mode with a
rb+ Open in binary read-write mode Same as r+
wb+ Open in binary read-write mode Same as w+
ab+ Open in binary read-write mode Same as a+
buffering 1 Indicates that there is line buffering, and the default means that the buffering mechanism provided by the system is used by default.
Return value: successfully returns the file operation object.
2. Read the file
read([size]) Function: To directly read the characters in the file. Parameters: if not given size parameter (the default is-1)or size If the value is negative, the file will be read until the end, given size Read at most the given number of characters (bytes). Return value: return the read content
Note: It is not recommended to read directly to the end of the file when the file is too large, and an empty string will be returned when the file is read to the end of the file.
readline([size]) Function: used to read a line in a file Parameters: if not given size parameter (the default is-1)or size If the value is negative, it means to read a line, given size Indicates to read at most specified characters (bytes). Return value: return the read content
readlines([sizeint]) Function: Read each line in the file as an item in the list Parameters: if not given size parameter (the default is-1)or size If the value is negative, the file will be read until the end, given size means read size The line where the character is located. Return value: return the read content list
The file object itself is also an iterable object, and each line of the file can be iterated in the for loop.
for line in f:
print(line)
3. Write to file
write(string) Function: Write text data or a string of binary data blocks to a file Parameters: the content to write If you need a newline, you need to add it yourself in the writing content\n
writelines(str_list) Function: Accepts a list of strings as parameters and writes them to a file. parameter: list of content to write
4. Close the file
After opening a file, we can operate the file through the file object. When the operation is over, use close() to close the object to prevent some misoperations and save resources.
file_object.close()
5. with operation
The with statement in python is used when accessing resources to ensure that no matter whether an error or exception occurs during processing, the
The "cleanup" operation specified by the line releases the accessed resources, such as automatic closing after reading and writing of files, automatic acquisition and release of locks in threads
wait.
The syntax of the with statement is as follows:
with context_expression [as target(s)]: with-body
You can use the with method without close(), because the object generated by with will be automatically processed after the statement block ends, so there is no need to close, but this file object can only be used in the with statement block.
with open('file','r+') as f:
f.read()
Notice
Add b to open the read and write requirements must be byte string
Any file can be opened in binary mode, but binary files will go wrong when opened in text mode for reading and writing
Code implementation: day4/file_open.py
"""
file_open.py
file open method
"""
# open a file
# f = open('a.py','r+') # requires the file to exist
# f = open('a.py','w') # File does not exist Create exists Clear
# f = open('a.py','a') # The file does not exist to create, there is an append
# f = open('a.py','rb') # After adding b, subsequent reading and writing are all operated on byte strings
"""
All files can be opened in binary mode(b)
However, binary format files cannot be opened in text mode(Subsequent read and write errors)
"""
f = open('mm.jpg','r')
# Read and write operations via f
# Close the file object
f.close()
Code implementation: day4/file_read.py
"""
file_read.py
File reading demo
"""
# open a file
f = open('Install.txt','r')
# read file
# data = f.read()
# print(data)
# read file contents in a loop
# while True:
# # read() will read an empty string if it reaches the end of the file
# data = f.read(1024)
# # Read to the end and break out of the loop
# if not data:
# break
# print(data)
# Read a line from a file
# data = f.readline(5)
# print(data)
# data = f.readline(5)
# print(data)
# read content form list
# data = f.readlines(20) # Read all lines where the first 20 bytes are
# print(data)
# read each row using a for loop
for line in f:
print(line) # Each iteration to a line of content
# closure
f.close()
Code implementation: day4/file_write.py
"""
file_write.py
file write operation
"""
# open a file
# f = open('a.py','a')
f = open('a.py','w')
# write operation
# f.write("hello ghost\n")
# f.write("Oh, what are you doing\n")
# Write each item in the list to the file separately
l = ['hello world\n','hello kitty\n']
f.writelines(l)
# f.close()
other operations
flush buffer
Buffering: The system automatically creates a buffer in the memory for each file being used. Data output from the memory to the disk must first be sent to the memory buffer, and then sent to the disk from the buffer. To read data from the disk, read a batch of data from the disk file into the memory buffer at a time, and then send the data from the buffer to the data area of the program.
Flush buffer conditions:
buffer is full
Program execution ends or the file object is closed
Line buffer encounters newline
Call the flush() function in the program
flush() After this function is called, a disk interaction will be performed, and the content in the buffer will be written to the disk.
Code implementation: day4/buffer.py
"""
buffer.py
buffer flush test
"""
# f = open('a.py','w',1) # line buffering
f = open('a.py','w')
while True:
data = input(">>")
if not data:
break
f.write(data + '\n')
f.flush() # flush buffer
f.close()
file offset
definition
When opening a file for operation, the system will automatically generate a record, which describes our series of operations on the file. This includes the file location for each operation. File read and write operations are performed from this location.
basic operation
tell() Function: Get file offset size
seek(offset[,whence]) Function:Move file offset position parameter: offset Represents the number of bytes moved relative to a position. Negative numbers mean forward movement, positive numbers mean backward movement. whence The default value of the base position is 0, which means counting from the beginning of the file, 1 means counting from the current position, and 2 means counting from the end of the file. The reference position must be 1 or 2 when the file must be opened in binary mode
Code implementation: day4/seek.py
"""
seek.py file offset test
"""
# Open the file offset with r,w at the beginning, and open the file offset with a at the end
f = open("mm.jpg",'rb+')
print(f.tell())
# f.write("Hello world")
#
# print(f.tell())
# Move backward 5 characters from the beginning
f.seek(1024,0)
f.write('Hello'.encode())
# data = f.read()
# print(data)
f.close()
file descriptor
definition
Each IO operation in the system will be assigned an integer as the number, which is the file descriptor of the IO operation.
get file descriptor
fileno() pass IO The object gets the corresponding file descriptor
file management function
get file size
os.path.getsize(file)
view file list
os.listdir(dir)
Check if the file exists
os.path.exists(file)
Determine file type
os.path.isfile(file)
Delete Files
os.remove(file)
Basics of Network Programming
The functions of computer network mainly include the realization of resource sharing and the rapid transmission of data information.
OSI seven layer model
Developing organization: ISO (International Organization for Standardization)
Role: Standardize network communication workflow
Application layer: Provide user services, specific functions are implemented by applications Presentation layer: data compression optimization encryption Session layer: establish user-level connections and select appropriate transport services Transport layer: Provides transport services Network layer: routing, network interconnection Link layer: perform data exchange and control the sending of specific data Physical layer: Provide hardware guarantee for data transmission, network card interface, transmission medium
advantage
- Created a unified workflow
- The divisions are clear, each performs its duties, and each step has a clear division of labor
- Reduced coupling between modules for easy development
Four-layer model (TCP/IP model)
Background: In actual work, engineers cannot fully operate according to the requirements of the seven-layer model, and gradually evolve into a four-layer model that is more in line with the actual situation
data transfer process
- At the sending end, the application program sends the message, adds the header information layer by layer, and finally sends the message packet at the physical layer.
- The sent message is transmitted through multiple nodes (switches, routers) and finally reaches the target host.
- The target host parses the first message packet layer by layer by the physical layer, and finally presents the message to the application program.
Network protocol
In the network data transmission, the regulations to be followed include what kind of data structure to establish, what kind of special signs, etc.
Network Basic Concepts
IP address
Function: Determine the network routing location of a host
View the local network address command: ifconfig
structure
IPv4 dotted decimal notation 172.40.91.185 The value range of each part is 0–255
IPv6 128-bit expands the address range
domain name
Definition: A name given to a web server address
Function: To facilitate memory and express a certain meaning
ping [ip] : Test whether it is connected to a host
port number (port)
Function: The port is part of the network address and is used to distinguish different network applications on the host.
Features: Application listening ports in a system cannot be repeated
Value range: 1 – 65535
1–1023 system application or public program listening port
1024–65535 Own ports
transport layer service
Connection-oriented transmission service (data transmission based on TCP protocol)
- Transmission characteristics: Provides reliable data transmission, reliability refers to no loss, no disorder, no error, no duplication during data transmission.
- Means of implementation: A data connection needs to be established before communication, and the connection should be disconnected normally after the communication ends.
Three-way handshake (connection establishment)
The client sends a message to the server requesting a connection
After the server receives the request, the reply message confirms that it can be connected
The client receives the reply and sends the final message to establish the connection
Four waves (disconnect)
The active party sends a message requesting to disconnect
After receiving the request, the passive party immediately responds, indicating that it is ready to disconnect
The passive party is ready, and sends a message again to indicate that it can be disconnected
The active party receives the confirmation and sends the final message to complete the disconnection
Applicable situation: There are clear requirements for the accuracy of data transmission, the data transmission file is large, and the reliability needs to be ensured. For example: web page acquisition, file download, email sending and receiving.
Connectionless transmission service (data transmission based on UDP protocol)
- Transmission characteristics: The reliability of transmission is not guaranteed, there is no connection and disconnection during the transmission process, and data can be sent and received freely.
- Applicable conditions: The network is poor, and the requirements for transmission reliability are not high. For example: online video, group chat, broadcasting
interview requirements
Let me introduce the OSI seven-layer model. What is the tcp/ip model?
What is the difference between tcp service and udp service?
What is the three-way handshake and four-way finger wave, and what is the process like?
socket socket programming
Introduction to sockets
- Socket: A technical means to realize network programming for data transmission
- Python implements socket programming: import socket
- Socket classification
Stream socket (SOCK_STREAM): transmit data in byte stream to realize tcp network transmission scheme. (connection oriented – tcp protocol – reliable – stream sockets)
Datagram socket (SOCK_DGRAM): transmit data in the form of datagram to realize the udp network transmission scheme. (connectionless – udp protocol – unreliable – datagram sockets)
tcp socket programming
Server process
- create socket
sockfd=socket.socket(socket_family=AF_INET,socket_type=SOCK_STREAM,proto=0) Function: create socket parameter: socket_family network address type AF_INET express ipv4 socket_type socket type SOCK_STREAM(streaming) SOCK_DGRAM(Datagram) proto Usually 0 to select the sub-protocol Return value: socket object
2. Binding address
Local address: 'localhost' , '127.0.0.1' website address : '172.40.91.185' Obtain address automatically: '0.0.0.0'
sockfd.bind(addr)
Function: Bind the local network address
Parameters: 2-tuple (ip,port) ('0.0.0.0',8888)
- set monitor
sockfd.listen(n) Function: Set the socket as a listening socket and determine the size of the listening queue Parameters: listener queue size
- Waiting for client connection requests to be processed
connfd,addr = sockfd.accept()
Function: Block waiting to process client requests
return value: connfd client connection socket
addr The client address to connect to
- messaging
data = connfd.recv(buffersize) Function : accept client message Parameters: the size of the maximum message received each time Return value: received content n = connfd.send(data) Function : Send a message Parameters: the content to send bytes Format Return value: the number of bytes sent
- close socket
sockfd.close()
Function: close socket
client process
- create socket
Note: Only sockets of the same type can communicate
- request connection
sockfd.connect(server_addr) Function: connect to server Parameters: tuple server address
- send and receive messages
Note: To prevent blocking at both ends, recv send should cooperate
- close socket
Code implementation: day5/tcp_server.py
"""
tcp_server.py tcp Socket server function flow
key code
Note: Pay attention to process and function usage
"""
import socket
from time import sleep
# Create a TCP socket
sockfd = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
# binding address
sockfd.bind(('0.0.0.0',8889))
# set monitor
sockfd.listen(5)
# Blocking waiting for a client connection
while True:
print("Waiting for connect...")
try:
connfd,addr = sockfd.accept()
print("Connect from",addr) # print connected clients
except KeyboardInterrupt:
print("server exit")
break
except Exception as e:
print(e)
continue
# send and receive messages
while True:
data = connfd.recv(5)
# If the connected client exits, recv will immediately return an empty string
if not data:
break
print(data.decode())
sleep(0.1)
n = connfd.send(b"Thanks#")
print("Send %d bytes"%n)
connfd.close()
# close socket
sockfd.close()
Code implementation: day5/tcp_client.py
"""
tcp_client.py tcp client process
key code
"""
from socket import *
# create tcp socket
sockfd = socket() # Default parameter -->tcp socket
# Connect to the server program
server_addr = ('172.40.91.143',8889)
sockfd.connect(server_addr)
# send receive message
while True:
data = input("Msg:")
# data is empty to exit the loop
if not data:
break
sockfd.send(data.encode()) # send byte string
data = sockfd.recv(1024)
print("Server:",data.decode())
# close socket
sockfd.close()
tcp socket data transmission characteristics
When one end of the tcp connection exits, if the other end is blocked in recv, recv will immediately return an empty string.
If one end of the tcp connection does not exist, if you still try to send it through send, a BrokenPipeError will occur
A listening socket can connect to multiple clients at the same time, and can also be connected repeatedly
Network send and receive buffer
- The network buffer effectively coordinates the sending and receiving speed of messages
- send and recv actually send and receive messages to the buffer, and recv will not block when the buffer is not empty.
tcp sticky packet
Reason: tcp is transmitted in byte stream, without message boundaries. A message sent multiple times is received once, and a sticky packet will be formed at this time.
Impact: If the content sent each time has an independent meaning, the receiving end needs to analyze it independently. At this time, the sticky packet will have an impact.
Approach
- Artificially adding message boundaries
- Control sending speed
UDP socket programming
Server process
- Create a datagram socket
sockfd = socket(AF_INET,SOCK_DGRAM)
- binding address
sockfd.bind(addr)
- messaging
data,addr = sockfd.recvfrom(buffersize) Function: receive UDP information Parameters: How many bytes to receive at most each time return value: data received content addr message sender address n = sockfd.sendto(data,addr) Function: send UDP information parameter: data sent content bytes Format addr target address Return value: the number of bytes sent
- close socket
sockfd.close()
client process
- create socket
- send and receive messages
- close socket
Summary: The difference between tcp socket and udp socket programming:
-
. Stream sockets transmit data in the form of byte streams, and datagram sockets transmit data in the form of datagrams
-
The tcp socket will have sticky packets, and the udp socket will not have sticky packets if there are message boundaries
-
The tcp socket guarantees the integrity of the message, the udp socket does not
-
The tcp socket relies on listen accept to establish a connection to send and receive messages, but the udp socket does not need it
-
tcp socket uses send, recv to send and receive messages, udp socket uses sendto, recvfrom
Code implementation: day6/udp_server.py
"""
udp_server.py udp Server
key code
"""
from socket import *
# Create a UDP socket
sockfd = socket(AF_INET,SOCK_DGRAM)
# binding address
server_addr = ('0.0.0.0',8888)
sockfd.bind(server_addr)
# Loop send and receive
while True:
data,addr = sockfd.recvfrom(1024)
print("received message:",data.decode())
sockfd.sendto(b'Thanks',addr)
# close socket
sockfd.close()
Code implementation: day6/udp_client.py
"""
udp_client.py udp client
key code
"""
from socket import *
# server address
ADDR = ('127.0.0.1',8888)
# create socket
sockfd = socket(AF_INET,SOCK_DGRAM)
# Send messages in a loop
while True:
data = input("Msg:")
if not data:
break
sockfd.sendto(data.encode(),ADDR)
msg,addr = sockfd.recvfrom(1024)
print("From server:",msg.decode())
sockfd.close()
socket socket properties
[1] sockfd.type socket type [2] sockfd.family socket address type [3] sockfd.getsockname() Get the socket binding address [4] sockfd.fileno() Get the socket's file descriptor [5] sockfd.getpeername() Get connection socket client address [6] sockfd.setsockopt(level,option,value) Function: Set socket options parameter: level option category SOL_SOCKET option Specific options value option value
Code implementation: day6/sock_attr.py
"""
Introduction to Socket Properties
"""
from socket import *
# create socket object
s = socket(AF_INET,SOCK_STREAM)
# Set port to reuse immediately
s.setsockopt(SOL_SOCKET,SO_REUSEADDR,1)
s.bind(('172.40.91.143',8888))
s.listen(3)
c,addr = s.accept()
print(s.type) # socket type
print(s.family) # address type
print(s.getsockname()) # binding address
print(s.fileno()) # file descriptor
print(c.getpeername()) # Get the address of the connection end
c.recv(1024)
struct module for data packaging
- Principle: Pack a set of simple data, convert it into bytes format and send it. Or parse a set of bytes format data.
- interface use
Struct(fmt) Function: generate structured objects parameter: fmt custom data structures st.pack(v1,v2,v3....) Function: Package and convert a set of data according to the specified format into bytes Parameters: the data to pack return value: bytes byte string st.unpack(bytes_data) Function: Will bytes The byte string is parsed according to the specified format Parameters: The byte string to parse Return value: parsed content struct.pack(fmt,v1,v2,v3...) struct.unpack(fmt,bytes_data)
Description: You can use the struct module to call pack unpack directly. At this time, the first parameter of these two functions is passed to fmt. Other usage functions are the same
Code implementation: day6/struct_recv.py
"""
use udp，Enter student information from the client, and write student information to a file on the server
Each student information is required to occupy one line
ID NAME AGE SCORE
"""
from socket import *
import struct
# Define data transfer format
st = struct.Struct('i28sif')
# Create socket to accept content
s = socket(AF_INET,SOCK_DGRAM)
s.bind(('0.0.0.0',8888))
# open a file
f = open('student.txt','a')
while True:
data,addr = s.recvfrom(1024)
# Convert information to byte data (1,b'lily',13,89.5)
data = st.unpack(data)
info = "%d %-10s %d %.1f\n"%data
f.write(info)
f.flush()
f.close()
s.close()
Code implementation: day6/struct_send.py
from socket import *
import struct
# Define data transfer format
st = struct.Struct('i28sif')
# Create socket to accept content
s = socket(AF_INET,SOCK_DGRAM)
# server address
ADDR = ('127.0.0.1',8888)
while True:
print("================================")
id = int(input("ID:"))
name = input("Name:").encode()
age = int(input("Age:"))
score = float(input("Score:"))
#data packing
data = st.pack(id,name,age,score)
s.sendto(data,ADDR)
HTTP transport
HTTP protocol (Hypertext Transfer Protocol)
- Purpose: Web page acquisition, data transmission
- features
Application layer protocol, transport layer uses tcp transmission
Simple and flexible, many languages have HTTP-specific interfaces
Stateless, the protocol does not record the transmission content
http1.1 supports persistent connections and enriches request types
- Web page request process
1. The client (browser) sends an http request to the server through tcp transmission
2. The server parses after receiving the http request
3. The server processes the request content and organizes the response content
4. The server sends the response content to the browser in http response format
5. The browser receives the response content, parses and displays it
HTTP request (request)
Request line: specific request category and request content
GET / HTTP/1.1 Request category Request content protocol version
Request class: Each request class means to do different things
GET : Get network resources POST : Submit certain information and get feedback HEAD : Get only response headers for network resources PUT : update server resource DELETE : delete server resources CONNECT TRACE : test OPTIONS : Get server performance information
Request header: further explanation and description of the request
Accept-Encoding: gzip
blank line
Request body: request parameters or submitted content
Code implementation: day6/http_test.py
"""
http Request a demo of the process
"""
from socket import *
# tcp socket (http-->tcp)
s = socket()
s.bind(('0.0.0.0',8000))
s.listen(3)
c,addr = s.accept()
print("Connect from",addr)
# get request
data = c.recv(4096)
print(data)
# return response
response = """HTTP/1.1 200 OK
Content-Type:text/html
Hello World
"""
c.send(response.encode())
c.close()
s.close()
Code implementation: day7/http_server.py
"""
httpserver v1.0
Basic requirements: 1. Get the request from the browser
2. Determine whether the request content is/
3. If yes, then the index.html send to browser
If not, tell the browser sorry
4. Pay attention to organization http Response format, judgment 200 or 404
"""
from socket import *
# Handle client requests
def request(connfd):
# Get HTTP requests directly
data = connfd.recv(4096)
# prevent browser disconnection
if not data:
return
# Fetch request content
request_line = data.decode().split('\n')[0]
info = request_line.split(' ')[1]
# Determine whether the request content is /
if info == '/':
with open('index.html') as f:
response="HTTP/1.1 200 OK\r\n"
response+="Content-Type:text/html\r\n"
response+="\r\n"
response+=f.read()
else:
response = "HTTP/1.1 404 Not Found\r\n"
response += "Content-Type:text/html\r\n"
response += "\r\n"
response += "<h1>Sorry....</h1>"
# send to browser
connfd.send(response.encode())
# Build tcp service
sockfd = socket()
sockfd.setsockopt(SOL_SOCKET,SO_REUSEADDR,1)
sockfd.bind(('0.0.0.0',8080))
sockfd.listen(5)
while True:
connfd,addr = sockfd.accept()
print("Connect from",addr)
request(connfd) # Specifically handle client requests
http response (response)
- Response format: response line, response header, blank line, response body
Response line: Feedback the basic response
HTTP/1.1 200 OK Version Information Response Code Additional Information
Response code:
1xx Prompt message, indicating that the request has been received 2xx successful response 3xx Response requires further action, redirect 4xx client error 5xx Server Error
Response header: a description of the response content
Content-Type: text/html
Response body: the body content information of the response