Recently, have had some scheduling tasks that need to execute on the website. This will cause high effort for this manually routine job.
For reducing the effort, I build an automation mechanism to replace the routine job, here will record the process for anyone who needs refer.
This article will use an example as a case to demonstrate how to build this automation process, and the following are this example of system processing:
Requirement
We join a website activity for getting the daily coin, and these tasks need manually access a website landing page, and log in to the OpenID with a Facebook login. After login, need to confirm the tasks button, if the button status is “Get coins”, will need to click and get coins, and if the status is “Check Info”, it means the daily tasks are already done.
Accuracy and precision are need execute the process at 04:00 am and should notify the final process to me.
Technical
For automation processing, here are some tech stacks I choose:
- Python: design the full system process
- Selenium: a tool for automation interaction with the website
- Chromium: to be a web driver
Environment settings
First, need to install Python:
sudo yum install python3
sudo yum install python3-pip
Install Chromium and relative package：
sudo yum install epel-release
sudo yum install chromium
Install Selenium library
pip3 install selenium
Download Chromium WebDriver：
Selenium needs a compatible version with Chromium and WebDriver。Can download WebDriver and de-compress Download page: https://sites.google.com/chromium.org/driver/downloads?authuser=0
Make sure the version of WebDriver you download matches the version of Chromium you have installed.
For example:
wget https://chromedriver.storage.googleapis.com/114.0.5735.90/chromedriver_linux64.zip
uzip chromedriver_linux64.zip
Now, Selenium and Chromium have been successfully installed on CentOS and the path to WebDriver has been set. We can use the Selenium Python library to control the Chromium browser.
Non-graphical Linux environment
Because my Linux host is not a real graphical environment, it may be necessary to set a virtual display (Virtual Display) to simulate a graphical interface. We can use Xvfb (X virtual frame buffer) or other similar tools to create a virtual display. This way, the Chromium browser will be able to run even without an actual monitor.
Install Xvfb with the following command:
sudo yum install xorg-x11-server-Xvfb
Setting environment variable:
For Xvfb to run correctly, some environment variables need to be set. Execute the following command in the terminal:
export DISPLAY=:99
Please note that:99 is the display number of the virtual display, you can adjust this value according to your needs.
Execute Xvfb:
Xvfb :99 -screen 0 1280x1024x24 &
This will start Xvfb on virtual display:99 with a resolution set to 1280x1024 and a color depth of 24 bits.
Next, the next step is to install Google Chrome
sudo dnf install google-chrome-stable
If Error: GPG check FAILED error occurs, then bypass GPG (this will bypass GPG verification, but be aware that this reduces security.)
sudo dnf install google-chrome-stable --nogpgcheck
Execute the Python program:
Next, We can execute the Python program and use Selenium to control the Chromium browser. Before executing a Python program, make sure the DISPLAY environment variable is set to the correct virtual display.
DISPLAY=:99 python main.py
Example
Here is a Python example using the Selenium module to achieve the goal:
# -*- coding: utf-8 -*-
import pytest
import time
import json
import logging
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from selenium.common.exceptions import NoSuchElementException, TimeoutException
class TestGetgift():
def setup_method(self, method):
webdriver_path = '/home/chromedriver'
self.driver = webdriver.Chrome(executable_path=webdriver_path)
self.vars = {}
def teardown_method(self, method):
self.driver.quit()
def test_getgift(self):
self.driver.get("https://example.com")
self.driver.set_window_size(1024, 768)
wait = WebDriverWait(self.driver, 10)
element = wait.until(expected_conditions.visibility_of_element_located((By.CSS_SELECTOR, 'img[alt="facebook"]')))
self.driver.find_element(By.CSS_SELECTOR, 'img[alt="facebook"]').click()
wait = WebDriverWait(self.driver, 10)
element = wait.until(expected_conditions.visibility_of_element_located((By.ID, 'email')))
self.driver.find_element(By.ID, "email").send_keys("YOUR EMAIL")
self.driver.find_element(By.ID, "pass").send_keys("YOUR PASSWORD")
self.driver.find_element(By.ID, "loginbutton").click()
time.sleep(2)
self.driver.get("https://example.com")
wait = WebDriverWait(self.driver, 10)
element = wait.until(expected_conditions.visibility_of_element_located((By.CSS_SELECTOR, '.mainPage_mainPage__container .swiper-slide-active')))
elem = self.driver.find_element_by_css_selector('.mainPage_mainPage__container .swiper-slide-active')
html = elem.get_attribute('outerHTML')
print(html)
try:
wait = WebDriverWait(self.driver, 10)
element = wait.until(expected_conditions.visibility_of_element_located((By.CSS_SELECTOR, '.mainPage_mainPage__container .swiper-slide-active .button_button')))
self.driver.find_element(By.CSS_SELECTOR, ".mainPage_mainPage__container .swiper-slide-active .button_button").click()
time.sleep(3)
//send notification
except TimeoutException:
logging.info("already get coin")
//send notification
logging.info("done")
if __name__ == "__main__":
test = TestGetgift()
test.setup_method(None)
test.test_getgift()
test.teardown_method(None)
Finally, establish a schedule
0 4 * * * DISPLAY=:99 /usr/bin/python3 ~/main.py >> ~/cron.log 2>&1
Error Handling
If install the package has the error message
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-3: ordinal not in range(128)
It means system default is ASCII, and can solve this problem with defined the
export LANG=en_US.utf8
ex