Tech lead Web Crawling
We are looking for an ambitious lead who can build solution which can scrape data without much developer intervention. This requires a high degree of automation.
Experience
- 8 to 12 years
Functional Area
- IT Software - Programming
Role Category
- Management
Employment Type
Permanent Job, Full Time
Working Hours
-
Mon to Fri 2 PM to 11 PM
Work Location
- Vadodara
- 6 to 8 years of experience in Web Scraping or Crawling through Scrapy, Selenium or other frameworks and related libraries (like beautifulsoup, puppeteer).
- Should be expert on latest version of Python.
- Should have very good experience on fetching data from multiple online sources, cleanse it and build APIs on top of it.
- Good understanding of data structure and algorithms, as well as how they affect system performance in real-world applications
- Sound knowledge in bypassing Bot Detection techniques.
- Web RestFul APIs / Microservices Development Experience.
- Think deeply about developing large scale scraping tools including data integrity, health and monitoring systems.
- Develop a deep understanding of our vast data sources on the web and know exactly how, when, and which data to scrape, parse and store.
- Work with SQL and NoSQL databases to store raw data.
- Develop frameworks for automating and maintaining constant flow of data from multiple sources.
- Good knowledge of distributed technologies, real-time systems of high throughput, low latency, and highly scalable systems.
- Work independently with little supervision to research and test innovative solutions.
- Should have a strong passion for coding.
- Must take quality, security and performance seriously.
- Ability to pair with other engineers and cross-team as needed.
- Excellent communication skills, including the ability to present effectively to both business and technical audiences at all levels of the organization.
Skills Required