Testers advanced must-see series "python automated testing tool selenium usage guide"

selenium is the most popular automated testing tool in web applications, which can be used for automated testing or browser crawlers. The official website address is: selenium . Compared with another web automated testing tool QTP, it has the following advantages:

Free open source and lightweight, different languages only need a small dependency package
Support multiple systems, including Windows, Mac, Linux
Support multiple browsers, including Chrome, FireFox, IE, safari, opera, etc.
Support multiple languages, including Java, C, python, c# and other mainstream languages
Support for distributed test case execution

python+selenium environment installation

First, you need to install the python (recommended 3.7+) environment, and then directly install the dependency package with pip install selenium.

In addition, you need to download the webdriver corresponding to the browser. Note that the downloaded driver version must match the browser version.

Firefox browser driver: geckodriver
Chrome browser driver: chromedriver
IE browser driver: IEDriverServer
Edge browser driver: MicrosoftWebDriver
Opera browser driver: operadriver

After downloading, you can add the driver to the environment variable, so that you don't need to manually specify the driver path when using it.

Start the browser with selenium

You can use the following code to start a Chrome browser in python, and then control the browser's behavior or read data.

from selenium import webdriver

# Start the Chrome browser, requiring that the chromedriver driver has been configured to the environment variable
# Putting the driver in the same folder as the current script also works
driver = webdriver.Chrome()

# Manually specify the driver path
driver = webdriver.Chrome(r'D:/uusama/tools/chromedriver.exe')

driver = webdriver.Ie()        # Internet Explorer browser
driver = webdriver.Edge()      # Edge browser
driver = webdriver.Opera()     # Opera browser
driver = webdriver.PhantomJS()   # PhantomJS

driver.get('http://uusama.com') # Open the page with the specified path

You can also set startup parameters during startup. For example, the following code implements adding a proxy at startup and ignores the https certificate verification.

from selenium import webdriver

# Create a chrome launch options object
options = webdriver.ChromeOptions()


options.add_argument("--proxy-server=127.0.0.1:16666")  # set proxy
options.add_argument("---ignore-certificate-errors")  # Set to ignore https certificate verification
options.add_experimental_option("excludeSwitches", ["enable-logging"])  # enable log

# Set the default path saved when the browser downloads files
prefs = {"download.default_directory": get_download_dir()}
options.add_experimental_option("prefs", prefs)
driver = webdriver.Chrome(options=options)

Some very useful startup options, options = webdriver.ChromeOptions() used below:

options.add_argument("--proxy-server=127.0.0.1:16666"): Set the proxy, which can be combined with mitmproxy to capture packets, etc.
option.add_experimental_option('excludeSwitches', ['enable-automation']): Set to bypass selenium detection
options.add_argument("---ignore-certificate-errors"): Set to ignore https certificate verification
options.add_experimental_option("prefs", {"profile.managed_default_content_settings.images": 2}): Set no request image mode to speed up page loading
chrome_options.add_argument('--headless'): set headless browser

selenium page load wait and detection

After using selenium to open the page, it cannot be operated immediately. It needs to wait until the loading of the pending page elements is completed. At this time, it is necessary to detect and wait for the loading of the page elements.

Use time.sleep() to wait

The easiest way is to use time.sleep() to wait for a certain period of time after opening the page. This method can only set a fixed time to wait. If the page is loaded in advance, it will be blocked in vain.

from time import sleep
from selenium import webdriver

driver = webdriver.Chrome()
driver.get('http://uusama.con')
time.sleep(10)
print('load finish')

Use implicitly_wait to set the maximum waiting time

In addition, you can use implicitly_wait to set the maximum waiting time. If the page is loaded within a given time or has timed out, the next step will be executed. This method will wait until all resources are loaded, that is, the loading chart in the browser tab bar does not turn before executing the next step. It is possible that the page elements have been loaded, but resources such as js or pictures have not been loaded yet, and you need to wait at this time.

It should also be noted that using implicitly_wait only needs to be set once, and it works for the entire driver life cycle. Whenever a page is loading, it will be blocked.

Examples are as follows:

from selenium import webdriver

driver = webdriver.Chrome()
driver.implicitly_wait(30)   # Set to wait up to 30 seconds
driver.get('http://uusama.com')
print(driver.current_url)

driver.get('http://baidu.com')
print(driver.current_url)

Use WebDriverWait to set wait conditions

Using WebDriverWait (selenium.webdriver.support.wait.WebDriverWait) can set the waiting time more accurately and flexibly. WebDriverWait can detect whether a certain condition is met at regular intervals within the set time, and proceed to the next step if the condition is met. If it is not satisfied after the set time, a TimeoutException will be thrown, and its method declaration is as follows:

WebDriverWait(driver, timeout, poll_frequency=0.5, ignored_exceptions=None)

The meanings of the parameters are as follows:

driver: browser driver
timeout: the maximum timeout time, the default is in seconds
poll_frequency: detection interval (step size), the default is 0.5 seconds
ignored_exceptions: ignored exceptions, even if the given exception is thrown during the call to until() or until_not(), it will not interrupt

WebDriverWait() is generally used with the until() or until_not() method, which means waiting to block until the return value is True or False. It should be noted that the parameters of these two methods must be callable objects, that is, the method name, which can be used in the expected_conditions module The method or the method of self-encapsulation.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions

driver = webdriver.Chrome()
driver.get("http://baidu.com")

# Judging whether the element with the id `input` has been added to the dom tree does not mean that the element must be visible. If it is located, it will return WebElement
element = WebDriverWait(driver, 5, 0.5).until(expected_conditions.presence_of_element_located((By.ID, "s_btn_wr")))

# When both implicitly_wait and WebDriverWait are set, take the largest waiting time between the two
driver.implicitly_wait(5)

# Determine whether an element has been added to the dom and is visible. Visible means that the element can be displayed and its width and height are greater than 0
WebDriverWait(driver,10).until(EC.visibility_of_element_located((By.ID, 'su')))

# Determine whether the element is visible, and return the element if visible
WebDriverWait(driver,10).until(EC.visibility_of(driver.find_element(by=By.ID, value='kw')))

Some methods commonly used by expected_conditions are listed below:

title_is: Determine whether the title of the current page is exactly as expected
title_contains: Determine whether the title of the current page contains the expected string
presence_of_element_located: Judging whether an element is added to the dom tree does not mean that the element must be visible
visibility_of_element_located: Determine whether an element is visible (the element is not hidden, and the width and height of the element are not equal to 0)
visibility_of: Do the same thing as the above method, except that the above method needs to be passed to the locator, and this method can directly pass the positioned element
presence_of_all_elements_located: Determine whether at least one element exists in the dom tree. For example, if there are n elements on the page whose class is 'column-md-3', then as long as there is 1 element, this method will return True
text_to_be_present_in_element: Determine whether the text in an element contains the expected string
text_to_be_present_in_element_value: Determine whether the value attribute in an element contains the expected string
frame_to_be_available_and_switch_to_it: Determine whether the frame can be switched in, if so, return True and switch in, otherwise return False
invisibility_of_element_located: Determine whether an element does not exist in the dom tree or is invisible
element_to_be_clickable: Determine whether an element is visible and enable d, so it is called clickable
staleness_of: wait for an element to be removed from the dom tree, note that this method also returns True or False
element_to_be_selected: Determine whether an element is selected, generally used in drop-down lists
element_selection_state_to_be: Determine whether the selected state of an element meets expectations
element_located_selection_state_to_be: Same as the above method, except that the above method passes in the positioned element, and this method passes in the locator

Check if the document is loaded

In addition, you can use driver.execute_script('return document.readyState;') == 'complete' to detect whether the document is loaded.

Note that the document loading is completed, which does not include the asynchronous loading ajax request to dynamically render the dom. This requires using WebDriverWait to detect whether an element has been rendered.

selenium element positioning and reading

find element

selenium provides a series of APIs to facilitate access to elements in chrome. These APIs all return WebElement objects or their lists, such as:

find_element_by_id(id): Find the first element matching id
find_element_by_class_name(): Find the first element matching class
find_elements_by_xpath(): Find all elements matching xpath
find_elements_by_css_selector(): Find all elements that match the css selector

In fact, you can look at the implementation source code in the WebDriver class. Its core implementation calls two basic functions:

find_element(self, by=By.ID, value=None): Find the first element matching the strategy
find_elements(self, by=By.ID, value=None): Find all elements matching the strategy

The by parameter can be ID, CSS_SELECTOR, CLASS_NAME, XPATH, etc. Here are a few simple examples:

Query the first element containing the text login through xpath: find_element_by_xpath("//*[contains(text(),'login')]")
Query all elements containing the class name refresh: find_elements_by_class_name('refresh')
Query the second row of the table form: find_element_by_css_selector('table tbody > tr:nth(2)')

dom element interaction

The element search results described above are WebElement objects. The commonly used APIs are:

element.text: returns the text content of the element (including the content of all descendant nodes), note that if the element display=none, it will return an empty string
element.screenshot_as_png: Screenshot of the element
element.send_keys("input"): element input box input input string
element.get_attribute('data-v'): Get data-v name attribute value, in addition to custom node attributes, you can also get attributes such as textContent
element.is_displayed(): Determine whether the element is visible to the user
element.clear(): clear element text
element.click(): Click on the element, if the element is not clickable, ElementNotInteractableException will be thrown
element.submit(): Simulate form submission

Find element failure handling

If the specified element cannot be found, NoSuchElementException will be thrown, and it should be noted that the element with display=none can be obtained, and all elements in the dom node can be obtained.

And when you actually use it, you should pay attention to some elements dynamically created by js code, which may need to be polled or monitored.

A method that checks for the existence of a specified element is as follows:

def check_element_exists(xpath):
    try:
        driver.find_element_by_xpath(xpath)
    except NoSuchElementException:
        return False
    return True

selenium interactive control

ActionChains action chain

webdriver simulates user operations through the ActionChains object, which represents an action link queue, and all operations will enter the queue in turn but will not be executed immediately until the perform() method is called. Its commonly used methods are as follows:

click(on_element=None): click the left mouse button
click_and_hold(on_element=None): Click the left mouse button without releasing it
context_click(on_element=None): click the right mouse button
double_click(on_element=None): Double click the left mouse button
send_keys(*keys_to_send): Send a key to the currently focused element
send_keys_to_element(element, *keys_to_send): Send a key to the specified element
key_down(value, element=None): Press a key on the keyboard
key_up(value, element=None): release a key
drag_and_drop(source, target): drag to an element and release
drag_and_drop_by_offset(source, xoffset, yoffset): drag to a certain coordinate and release
move_by_offset(xoffset, yoffset): Move the mouse from the current position to a certain coordinate
move_to_element(to_element): Move the mouse to an element
move_to_element_with_offset(to_element, xoffset, yoffset): How much distance to move to an element (upper left corner coordinates)
perform(): execute all actions in the chain
release(on_element=None): Release the left mouse button at an element position

Simulate mouse events

The following code simulates mouse movement, click, drag and other operations. Note that you need to wait for a certain period of time during the operation, otherwise the page will not have time to render.

from time import sleep
from selenium import webdriver
# Introducing the ActionChains class
from selenium.webdriver.common.action_chains import ActionChains

driver = webdriver.Chrome()
driver.get("https://www.baidu.cn")
action_chains = ActionChains(driver)

target = driver.find_element_by_link_text("search")
# Move the mouse to the specified element and click
action_chains.move_to_element(target).click(target).perform()
time.sleep(2)

# You can also directly call the click method of the element
target.click()
time.sleep(2)

# Move the mouse to (10, 50) coordinates
action_chains.move_by_offset(10, 50).perform()
time.sleep(2)

# The mouse moves to the distance element target(10, 50)
action_chains.move_to_element_with_offset(target, 10, 50).perform()
time.sleep(2)

# Mouse dragging, dragging one element to another
dragger = driver.find_element_by_id('dragger')
action.drag_and_drop(dragger, target).perform()
time.sleep(2)

# You can also use click -> move to achieve drag and drop
action.click_and_hold(dragger).release(target).perform()
time.sleep(2)
action.click_and_hold(dragger).move_to_element(target).release().perform()

Simulate keyboard input events

Simulate keyboard events through send_keys, commonly used are:

send_keys(Keys.BACK_SPACE): delete key (BackSpace)
send_keys(Keys.SPACE): space bar (Space)
send_keys(Keys.TAB): tab key (Tab)
send_keys(Keys.ESCAPE): Back key (Esc)
send_keys(Keys.ENTER): Enter key (Enter)
send_keys(Keys.F1): keyboard F1
send_keys(Keys.CONTROL,'a'): select all (Ctrl+A)
send_keys(Keys.CONTROL,'c'): copy (Ctrl+C)
send_keys(Keys.CONTROL,'x'): Cut (Ctrl+X)
send_keys(Keys.CONTROL,'v'): Paste (Ctrl+V)

Example: Locate the input box and enter content

# Input box input content
driver.find_element_by_id("kw").send_keys("uusamaa")

# Simulate carriage return to delete a character a of multiple input
driver.find_element_by_id("kw").send_keys(Keys.BACK_SPACE)

Alert Box Handling

It is used to handle the alert box popped up by calling alert.

driver.switch_to_alert(): Switch to the alert box
text: returns the text information in alert/confirm/prompt, for example, if js calls alert('failed'), it will get the failed string
accept(): accept the existing alert box
dismiss(): close the existing warning box
send_keys(keysToSend): Send text to the alert box

selenium browser control

Basic common api

Some very useful browser control APIs are listed below:

driver.current_url: Get the url of the current active window
driver.switch_to_window("windowName"): Move to the specified tab window
driver.switch_to_frame("frameName"): Move to the iframe with the specified name
driver.switch_to_default_content(): Move to the default text content area
driver.maximize_window(): maximize the browser display
driver.set_window_size(480, 800): set the browser width to 480 and height to 800 for display
driver.forword(), driver.back(): browser forward and backward
driver.refresh(): Refresh the page
driver.close(): close the current tab
driver.quiit(): close the entire browser
driver.save_screenshot('screen.png'): Save screenshot of the page
driver.maximize_window(): maximize the browser display
browser.execute_script('return document.readyState;'): execute js script

selenium reads and loads cookie s

Use get_cookies and add_cookie to cache cookies locally, and then load them at startup, so that the login status can be preserved. The implementation is as follows

import os
import json
from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.baidu.cn")

# read all cookie s and save to file
cookies = driver.get_cookies()
cookie_save_path = 'cookie.json'
with open(cookie_save_path, 'w', encoding='utf-8') as file_handle:
    json.dump(cookies, file_handle, ensure_ascii=False, indent=4)

# Read cookie from file and load to browser
with open(cookie_save_path, 'r', encoding='utf-8') as file_handle:
    cookies = json.load(file_handle)
    for cookie in cookies:
        driver.add_cookie(cookie)

selenium opens a new tab window

Using driver.get(url) will open the specified connection in the first tab window by default, and a new tab window will also be opened when clicking the _blank link in the page.

You can also use the following method to manually open a tab window of a specified page. Note that after opening a new window or closing it, you need to manually call switch_to.window to switch the currently active tab window, otherwise NoSuchWindowException will be thrown.

from selenium import webdriver

driver = webdriver.Chrome()
driver.get("https://www.baidu.cn")

new_tab_url = 'http://uusama.com'
driver.execute_script(f'window.open("{new_tab_url}", "_blank");')
time.sleep(1)

# Note: You must call switch_to.window to manually switch the window, otherwise you will not find the tab view
# Focus to the newly opened tab page, then close
driver.switch_to.window(driver.window_handles[1])
time.sleep(2)
driver.close()   # close current window

# Manually return to the original tab page
driver.switch_to.window(driver.window_handles[0])
time.sleep(1)

In addition to using execute_script, you can also create a new tab window by simulating the button to open a new tab page:

driver.find_element_by_tag_name('body').send_keys(Keys.CONTROL + 't')
ActionChains(driver).key_down(Keys.CONTROL).send_keys('t').key_up(Keys.CONTROL).perform()

Some problem records of selenium

Get the text content of hidden elements

If an element is hidden, that is, display=none, although the element can be found through find_element, the text content of the element cannot be obtained by using the element.text attribute, and its value is an empty string. At this time, the following can be used Way to get:

element = driver.find_element_by_id('uusama')
driver.execute_script("return arguments[0].textContent", element)
driver.execute_script("return arguments[0].innerHTML", element)

# Correspondingly, hidden elements can also be set to non-hidden
driver.execute_script("arguments[0].style.display = 'block';", element)

Browser crash WebDriverException exception handling

For example, if a page runs for a long time in Chrome, an Out Of Memory error will occur. At this time, WebDriver will throw a WebDriverException. Basically all APIs will throw this exception. At this time, it needs to be caught and handled specially.

My processing method is to record some basic information of the page, such as url, cookie, etc., and write it to the file regularly. If the exception is detected, restart the browser and load data such as url and cookie.

selenium crawls page request data

There are ways to obtain page requests through driver.requests or by parsing logs on the Internet, but I don't think it works very well. Finally, use the mitmproxy proxy to capture packets, and then fill in the proxy when starting selenium to achieve.

proxy.py is a custom proxy request processing packaged on the basis of mitmproxy, and its code is as follows:

import os
import gzip
from mitmproxy.options import Options
from mitmproxy.proxy.config import ProxyConfig
from mitmproxy.proxy.server import ProxyServer
from mitmproxy.tools.dump import DumpMaster
from mitmproxy.http import HTTPFlow
from mitmproxy.websocket import WebSocketFlow


class ProxyMaster(DumpMaster):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def run(self, func=None):
        try:
            DumpMaster.run(self, func)
        except KeyboardInterrupt:
            self.shutdown()


def process(url: str, request_body: str, response_content: str):
    # Packet capture request processing, where data can be dumped and parsed
    pass


class Addon(object):
    def websocket_message(self, flow: WebSocketFlow):
        # Listen to websockt requests
        pass

    def response(self, flow: HTTPFlow):
        # Avoid saving the flow all the time, causing the memory usage to soar
        # flow.request.headers["Connection"] = "close"
        # Listen to the http request response, and get the request body and response content
        url = flow.request.url
        request_body = flow.request
        response_content = flow.response

        # If the return value is compressed content, it needs to be decompressed
        if response_content.data.content.startswith(b'\x1f\x8b\x08'):
            response_content = gzip.decompress(response_content.data.content).decode('utf-8')
        Addon.EXECUTOR.submit(process, url, request_body, response_content)


def run_proxy_server():
    options = Options(listen_host='0.0.0.0', listen_port=16666)
    config = ProxyConfig(options)
    master = ProxyMaster(options, with_termlog=False, with_dumper=False)
    master.server = ProxyServer(config)
    master.addons.add(Addon())
    master.run()


if __name__ == '__main__':
    with open('proxy.pid', mode='w') as fin:
        fin.write(os.getpid().__str__())
    run_proxy_server()

In the process of using mitmproxy, proxy.py will have the problem of soaring memory usage over time. github's issue area Some people have also encountered it, and some say that it is because the http connection keep-alive=true request will always be saved and will not be released, resulting in more requests and more memory usage, and then by adding flow.request.headers["Connection"] = "close" To manually close the connection, I have some relief after adding it, but it still cannot be solved fundamentally.

Finally, record the proxy process by writing proxy.pid, and then restart proxy.py regularly with another program to solve the memory leak problem.

write at the end

If you think the article is not bad, please like, share, and leave a message, because this will be the strongest motivation for me to continue to output more high-quality articles!

People who read this article feel that my understanding is wrong, and welcome comments and discussions~

You can also join the group chat below to communicate with fellow masters

Tags: Python Selenium Testing

Posted by Birmingham on Wed, 29 Mar 2023 07:42:04 +1030

Programs Wiki