Web Crawler Engineer

Job Responsibilities

1. Responsible for web data scraping, cleaning, parsing, and storage.

2. Proficient in Python programming language, familiar with JavaScript, and experienced in building backend services with FastAPI.

3. Skilled in using databases, including Redis, PostgreSQL, and MySQL.

4. Experience with Docker for service deployment.

5. Familiar with various scraping technologies, including anti-scraping, proxies, and CAPTCHA handling.

6. Proficient in using regular expressions and XPath to extract information from structured and unstructured data.

7. Knowledgeable about common anti-scraping mechanisms and skilled in various anti-scraping techniques, including but not limited to: hook, JS protection, obfuscation, and reverse analysis.

8. Participate in the design and optimization of distributed web scraping systems, familiar with collection system scheduling strategies, real-time monitoring of scraper progress, and alert feedback.

Job Requirements

1. Programming skills: Proficient in Python, familiar with JavaScript, and able to use FastAPI to build backend services.

2. Database: Skilled in using PostgreSQL/MySQL for data storage and Redis for caching and deduplication.

3. Proficient in XPath/regular expression parsing of structured and unstructured data.

4. Knowledgeable in building proxy IP pools, request disguise, and dynamic rendering techniques.

5. Experience in anti-scraping techniques (Hook technology, JS reverse engineering, code obfuscation analysis, etc.) is preferred.

6. Experience with reverse engineering of algorithms such as Ruishu and Guomi is a plus.

7. Proficient in using Docker for containerized service deployment.

Employment Type

Full-time