Job responsibilities: Responsible for data crawling, including static web pages, dynamic web pages (JS rendering), API interface data, etc; Handle anti crawling strategies such as User Agent impersonation, proxy pooling, captcha bypass, cookie encryption, body parameter encryption, etc., to improve crawling success rate; Analyze webpage data and extract information using techniques such as XPath, CSS selectors, regular expressions, etc; Store and crawl data to databases such as MySQL, MongoDB, Redis, Selectdb, etc; Write data cleaning and deduplication related code to improve data quality; Monitor the running status of crawlers, optimize crawling strategies, and ensure the stability of data crawling;