Custom scrapers: what is worth automating, and what is not

Most businesses collect data by hand and never call it a job. Someone checks competitor prices every Monday morning. Someone else copies new listings into a spreadsheet. A third person pulls the same numbers off the same three websites before every management meeting. None of it appears on anyone's job description, and all of it costs real hours every week.

A custom scraper is just software doing that collection on a schedule. It is not exotic. It is also not always the right answer. Here is how we decide, what the data gets used for once it exists, and the compliance question a serious buyer asks first.

Start with the boring test

Before anything technical, three questions decide whether a scraper is worth building at all.

1. 1Is someone already doing this by hand, repeatedly? A weekly task is a candidate. A quarterly task usually is not, because the maintenance costs more than the hours you save.
2. 2Does the answer change often enough to matter? Data that shifts daily rewards automation. Data that shifts once a year does not.
3. 3Would fresher data change a decision? If you would price differently, restock differently or call a customer sooner, the collection has a value you can point at. If it just fills a slide, be honest about that.

A task that fails all three does not need automating. It needs dropping. We have talked clients out of scrapers on exactly these grounds, because a system nobody acts on is an ongoing cost with no return.

What the data is actually used for

Scraping is a means, not a purpose. The projects that work have an obvious use waiting for the output on day one.

• Competitor and market pricing. Publicly listed prices for comparable products, collected on a schedule, so pricing decisions are based on this week rather than on a memory of last quarter.
• Availability and stock signals. Retailers and parts suppliers use this to spot when a competitor runs dry, and when to promote what they do have.
• Your own listings. Businesses with presence across dozens of directories and marketplaces use scrapers to check their own details are still correct, which is a real problem once you pass twenty listings.
• Lead sourcing. Building a pipeline of companies that fit your customer profile, covered in more depth in lead generation automation.
• Internal reporting. Pulling numbers from tools that have no export worth using, and landing them in one place your team already looks at.
• Migration. A one-off extraction of a few thousand pages of catalogue or content from a platform you are leaving. Different job from the recurring ones, and often quicker.

The compliance question, answered honestly

Every serious buyer asks whether this is allowed, and a shrug is not an acceptable answer. The honest version is that it depends on what you collect, where it is published, and how you behave while collecting it. Those are all things you control.

• Public pages only. Content anyone can reach without an account or a paywall is a completely different proposition from content behind a login you were not given.
• Read the terms of service. Many sites state plainly what automated access they permit. Some allow it, some restrict it, some offer a data licence if you ask. That document decides the project, so it gets read before we quote.
• Respect robots.txt and a sane request rate. A well built scraper is slower than a person and puts less load on a site than one enthusiastic customer. Anything that degrades service for other visitors is badly built, whatever else you think of it.
• Personal data is its own category. Names, contact details and anything identifying a person carry obligations under UK and EU data protection law regardless of whether the page was public. That is a legal question, not a technical one.
• Prefer the official route. If there is an API, a feed, a bulk export or a licensing agreement, take it. It is more stable, usually cheaper to maintain, and removes the argument entirely.

When an off the shelf tool is the better answer

Custom is not automatically better. It is better when nothing existing fits, and it is a waste when something does.

Why scrapers break

Every scraper is a dependency on somebody else's website, and you have no say in when that website changes. Anyone who tells you their scraper will not break is selling you something. The realistic goal is that it breaks loudly, rarely, and gets fixed quickly.

• Markup changes. A redesign, or even a small class name change, and your selectors point at nothing.
• Navigation changes. Pagination becomes infinite scroll, filters move into a modal, a listing page starts loading its content later.
• Tightened limits. The site slows automated traffic or asks it to stop. The correct response is to slow down, reduce scope, or approach them about access. Not to escalate.
• Silent partial failure. The worst one by a distance. The job runs, writes half the usual rows and reports success, and nobody notices for three weeks.
• Downstream drift. The scraper is fine and the spreadsheet or database it feeds has quietly changed shape, so everything lands in the wrong column.

Unattended means monitored

These systems usually run on a VPS or a cloud server rather than on somebody's laptop, because the whole point is that they work while nobody is watching. That only holds if the setup treats failure as normal.

1. 1Scheduling. A cron job or a queue with a defined window, so runs do not overlap and do not all fire at once.
2. 2Retries with backoff. Transient errors are common. A retry after a pause fixes most of them without anyone waking up.
3. 3Sanity checks on the output. If yesterday returned 4,000 rows and today returns 12, that is a failure even though nothing threw an error.
4. 4Alerting to somewhere people look. Email, Slack, or a dashboard your team already opens. An alert in a log file nobody reads is decoration.
5. 5Raw responses kept for a short window. When something breaks, having the page as it was on the day turns a guessing game into a five minute fix.
6. 6A named owner. One person who gets the alert. Automation with no owner rots quietly.

This is the part that separates a script from a system, and it is the part most cheap scraper jobs skip. Data that stopped updating three weeks ago looks exactly like data that is current, right up until someone makes a decision on it.

How we approach a scraping project

We start with the sources and the terms attached to them, agree what the output has to look like and where it lands, then build, schedule and monitor it. Ongoing maintenance is quoted separately and honestly, because target sites change and pretending otherwise helps nobody.

If you have a collection task eating hours every week, tell us what it is and we will tell you whether it is worth automating, including when the answer is no. See web scraping and browser automation or AI workflow automation, message us on WhatsApp, or book a free 15-minute call.