Web Mining & Web Harvesting

Over the last few years, the World Wide Web has become a significant source of information and simultaneously a popular platform for business. Web mining can define as the method of utilizing data mining techniques and algorithms to extract useful information directly from the web, such as Web documents and services, hyperlinks, Web content, and server logs. The World Wide Web contains a large amount of data that provides a rich source to data mining. 

However, to extract public data from the web isn't an easy thing as it appears. Many customers found the crawlers they wrote keep broken due to website changes. Also many other people found that the website gets more and more tough to crawl and their crawlers can easily get blocked. This results into the cost of extracting public data getting higher and higher.  

At barkingdata, we mange thousands of crawlers  for client at low cost and affordable price. Customers are 100% freed up from dealing with blocking, captchas, proxies, server infrastructures, maintenances . Over the past 5 years, we have delivered over 5000 datasets to our customers and every week we are delivering over 100 million rows of data.

Barkingdata not only provides web harvesting capabilities, but also large dataset processing capabilities using distributed big data infrastructure. Many of our customers don't have the infrasture and the talents sets to manage and process large amount of data. Dataset beyond this level is hard to manage with tradational relational databases like MySQL, Sybase, SQL Server. Query billions of rows of dada with traditional database can be extremely slow.  

What It Costs to Develop and Manage Your Own Crawlers?

If you still perfer to manage crawlers on your own, please be aware that the following costs needs to be managed in order to extract data consistently and in an on-going basis.

Server costs

Maintaining in house scaping obviously requires servers for the crawling operation, crawling the data, and parsing it. It also requires setting up a complex load-balancing system, setting up an autoscale solution should also be considered as your crawling requirements may vary during the day/week. If you rely on cloud services like AWS or Digital ocean you should also factor in outbound traffic costs.

Proxy costs

Web harvesting and proxies go hand in hand, running a web crawler without good proxies is like owning a car without gas in its tank. Even if you rely on advanced web crawling tools like Puppeteer, without proxies websites would quickly figure out you are not a real user, and they would likely block your IP address, thus preventing you from accessing them in the future.

There are 2 kinds of IP addresses you should use when using proxies for web harvesting: Residential IPs – Residential proxies usually have a higher success rate, but they might be slower, tend to disconnect and significantly more expensive.

Maintenance

Finally, there is the matter of maintenance to consider, and that doesn’t simply include server maintenance or recurring proxy subscription payments. As you likely know, websites push updates all the time, and when a change like this occurs, the crawler needs to be updated. When extracting a webpage for data, a reference to some element in the page’s html needs to be defined. When extracting a few fields from a page these references are likely to be changed over time. Some websites even go the extra mile and place in anti-bot tools which update on a regular basis.

Data Accuracy and Monitoring

When using an in-house harvesting solution, you should also build a monitoring system that notifies you when things go wrong. Pages can change, IP Addresses can be blocked, and the crawlers you wrote might not cover all of the cases when pages vary. This means investing a large amount of resources in validating the extracted data and monitoring the quality of your web crawler.