Senior Backend Engineer (Data Extraction)

Job Summary We are looking for a skilled and motivated Senior Engineer to join our forward-thinking team. The ideal candidate is passionate about building cutting-edge AI solutions, has a strong technical foundation, and excels in collaborative environments. As an Senior Engineer, you will play a key role in designing, developing, and maintaining advanced data products and AI-powered applications. You'll work closely with cross-functional teams-including Data Scientists, Analytics Engineers, Front-End Developers, and ML Engineers-to deliver scalable and impactful solutions.

Now, if you were to come on board as our Senior Engineer, we'd ask you to do the following:

• Design and implement scalable, fault-tolerant systems capable for crawl, scrape, extract, and normalize data from diverse web sources at massive scale.
• Build robust pipelines to handle structured and semi-structured data seamlessly.
• Analyze and fine-tune crawling strategies to maximize data coverage, freshness, and quality.
• Conduct research to stay ahead of the curve in large-scale web acquisition and processing techniques.
• Take ownership of end-to-end engineering projects in data acquisition, including web crawling, data ingestion, and search.
• Continuously improve reliability, observability, and performance of data systems and tooling over time.
• Partner with cross-functional teams to prioritize crawling targets that align with key business needs.

Think you have what it takes to be our Senior Engineer? Here's how we'll know:

• 3+ years of industry experience working on large-scale, complex web scraping projects.
• Strong background as a backend or full-stack software engineer.
• Deep understanding of how to crawl the internet at scale.
• Strong programming skills, especially in Python and Go.

Bonus Points If You Have:

Expertise in internet-scale technologies such as DNS, networking, CDNs, WAFs, CAPTCHAs, proxies, and reverse proxies.

Experience with event-driven architectures.

Experience building observability and monitoring tools that support both engineers and non-technical stakeholders.

Exposure to large language models (LLMs) and their applications.