Web Crawler Engineer

Job Responsibilities

1. Responsible for web data scraping, cleaning, parsing, and storage.
2. Proficient in Python programming language, familiar with JavaScript, and experienced in building backend services with FastAPI.
3. Skilled in using databases, including Redis, PostgreSQL, and MySQL.
4. Familiar with deploying services using Docker.
5. Knowledgeable in various scraping technologies, including anti-scraping, proxies, and CAPTCHA handling.
6. Proficient in using regular expressions and XPath to extract information from structured and unstructured data.
7. Mastery of common anti-scraping mechanisms and expertise in various anti-scraping techniques, including but not limited to: hooking, JS protection, obfuscation, and reverse analysis.
8. Participate in designing and optimizing distributed web scraping systems, familiar with scheduling strategies for collection systems, real-time monitoring of scraping progress, and alert feedback.

Job Requirements

1. Programming skills: Proficient in Python, familiar with JavaScript, and able to use FastAPI to build backend services.
2. Database: Skilled in using PostgreSQL/MySQL for data storage and Redis for caching and deduplication.
3. Proficient in XPath and regular expressions for parsing structured and unstructured data.
4. Knowledge of building proxy IP pools, request disguise, and dynamic rendering techniques.
5. Experience in anti-scraping techniques (Hook technology, JS reverse engineering, code obfuscation analysis, etc.) is preferred.
6. Candidates with experience in reverse engineering of cryptographic algorithms are preferred.
7. Proficient in using Docker for containerized service deployment.

Employment Type

Full-time