27th Campus Recruitment - Web Crawler Engineer (J14414)

Responsibilities

1. Responsible for the collection development, structuring, and possible information extraction work for various target data dimensions based on data quality goals and requirements.
2. Fully understand the meaning and implementation methods of data quality quantitative indicators, actively maintain communication and coordination with the algorithm, cleaning, and data product teams to jointly promote the improvement of data quality.
3. Work with a mentor/team leader to explore and tackle relevant cutting-edge technologies and promote their implementation.

Requirements

1. Solid network foundation: Proficient in HTTP/HTTPS protocol stack, deeply understand Headers/Cookies/Session mechanisms, status code semantics, redirection strategies, etc. Familiar with TCP/IP protocol family and capable of using Wireshark/tcpdump for packet capture and anomaly analysis.
2. Master the basics of web front-end, familiar with browser rendering principles, and proficient in using browser DevTools for comprehensive debugging, including but not limited to: network request analysis, locating key parameter generation logic, breakpoint debugging, call stack tracing, DOM/event listener review, JS injection, etc.
3. Proficient in one or more programming languages such as Python/C/C++/C#/Java, with a preference for Python.
4. Familiar with the Linux operating environment, basic shell commands, process monitoring, log analysis, etc., and understand Docker/K8s container deployment.
5. Have certain experience in web page parsing and PDF/document parsing.
6. Have practical exploration or in-depth thinking on the combination of AI and traditional crawler technology, with a strong interest or practice in AI Agent-driven new crawler frameworks.
7. Strong sense of responsibility, good communication skills, and collaboration ability.

Preferred Qualifications

1. Experience in Android/iOS APP reverse engineering is preferred, such as proficiency in using Frida/Xposed tools.
2. Experience in AST code restoration is preferred.
3. Experience using mainstream AI tools (such as Codex, Claude Code, etc.), with the ability and experience to "guide AI" to assist in solving complex technical problems such as reverse engineering and algorithm restoration is preferred.