Case study
More than thirty bespoke scrapers, built and maintained over six years
Data extraction across automotive databases, real estate platforms and product catalogues, written per target and kept alive as those targets changed.
- Client
- Teoalida
- Sector
- Data research
- Engagement
- 2018 to 2024
- Delivered
- 30+ scrapers
The constraint
Teoalida sells research datasets. Before any of it can be sold it has to be collected, and it comes from sources that were never designed to be read by anything but a browser. Automotive specification databases, property listings, product catalogues, each with its own structure and its own idea of how much traffic is polite.
A generic scraping tool does not survive this. Every target needs its own handling for page structure, pagination, session state and whatever it does when it decides you are not a person. And the targets do not hold still. A scraper that worked last quarter breaks when the site is redesigned.
The approach
One scraper per target, written for that target. Over six years that came to more than thirty of them, in C# and Python depending on what the job needed. C# where the work was heavy and long-running, Python where the page was awkward and needed a real browser driving it.
The interesting part of this work is not the extraction, it is everything around it. Retrying without hammering the source. Backing off when asked. Validating what comes back, because a scraper that silently returns half a dataset is worse than one that fails loudly. And maintenance: these ran for years, which means most of the effort went on keeping them working rather than writing them.
What that involved
Core
- C# with .NET Framework and .NET Core
- Python with BeautifulSoup, Scrapy and Selenium
- XPath and CSS selectors
- JSON and XML parsing and transformation
- HTTP request handling and session management
Handling the hard parts
- Multi-threaded extraction
- JavaScript-rendered page handling
- Rate limiting and polite crawling
- Error handling and retry logic
- Data validation and cleaning pipelines
Where we draw the line
We build the technical solution to the client's specification. Responsibility for the legality of any data collection, and for compliance with the terms of service of the sources involved, sits with the client. That was agreed at the outset and it is how we still approach this kind of work.
The result
Six years of continuous work for one client, which is the part we would point at rather than the scraper count. Nobody keeps renewing an arrangement like this unless the output is reliable, and reliability in scraping is almost entirely about what happens after launch.
Six years of writing the same class of tool teaches you which parts are always the same. Those parts became Simple Web Scraper, a hosted version of this work that anyone can point at a page, rather than something we build one at a time.
The same client later asked us to solve their mapping problem, which became the property map engine and eventually ByteRemix Maps. Two of our products started as one client's problem.
Work with us
Need data that only exists on someone else's page?
Tell us what you need collected and how often.