May 10, 2023
web scraper, crawling, extension, web scraping
It's time for our latest feature release article, and we can't wait to show you what we've been working on!
October 25, 2022
Web Scraper Cloud, Patterns, data visualisation
Below, we’ll describe what Patterns is and walk through the set up for an app in Patterns that uses Web Scraper Cloud to acquire data and scan for product price differences on e-commerce websites. This app is a simple boilerplate template that can be extended to provide a more advances functionality.
September 09, 2022
web scraping, release
For the last few months, we have been working towards the Web Scraper 0.6.5 release and finally, it is here!
June 28, 2022
JQuery selectors, web scraper, Scraping public data, scraper, CSS selectors
Selectors when it comes to CSS and jQuery refer to code you can use to interact with specific elements in the DOM. They are particularly useful when you want to style or act upon certain elements of the document without altering other elements. Through a careful use of selectors, you can manipulate a page exactly as intended or isolate the specific data you’re looking to scrape.
May 02, 2022
data analysis, web scraper, amazon athena
Amazon Athena lets you use SQL to interactively query any data you have within an Amazon S3 Bucket. With Web Scraper Cloud, you can automatically sync web scraping job results to an Amazon S3 Bucket where you can then use Athena to quickly query this data with little additional overhead. In this article, we’re going to guide you through syncing Web Scraper Cloud with an S3 bucket and using Athena to generate and perform calculations on table data. While we’re covering a specific use case, you can adapt this same process to any other data set to gather insights.
March 21, 2022
web scraper, cheerio, web scraping, headless
In this article, we’re going to specifically cover how to set up a headless scraper with Cheerio and spoof browser headers to access a wider range of web pages.
March 03, 2022
web scraper, GDPR, data extraction, linkedin, Scraping public data, web scraping, facebook
We’ve all been hearing about the ongoing cases of companies failing lawsuits against web scraping activities. Worth mentioning are the cases of LinkedIn vs. HiQ on public data scraping and the case of Facebook Inc. vs. BrandTotal Ltd. on scraping data from Facebook. What is more interesting is that in one case it was ruled in favour of the data scraper, but in the other - of the company whose platform is being scraped. Seeing such cases with different outcomes raises questions, especially due to the recent General Data Protection Regulation (GDPR), on how lawful web scraping of publicly available data actually is?
September 01, 2021
Feature, Web Scraper Cloud, Tutorial, Pagination, release
The moment has come that we are ready to release the most requested feature of all time! Creating pagination correctly has been one of the most frequently reported problems; however, that is about to change. The pagination selector is here and so are other additional updates that will make scraping with Web Scraper easier.
August 02, 2021
For discovery and greater understanding, questions are a must. Therefore, we have collected the most frequently asked questions by the public about different web scraping topics.
June 14, 2021
Google, Google Sheets, Feature, Visualization, Web Scraper Cloud, release
Most people could agree that observing a picture, a simple graph, or a pie chart is easier and more efficient than scrolling through blocks of text or trying to parse out useful insights from looking at numerous columns in a spreadsheet. This common knowledge is the basis of why visualization is so important.
Also, in a rapidly changing world, people seek ways of becoming more efficient, ways of increasing the time spent focusing on the insights, rather than pouring endless hours and effort into collecting the necessary assets to gain those end results that determine so much. This is why automatization nowadays is a key topic.
June 01, 2021
Just like a vehicle tire leaves an imprint in the mud which can be washed away only by a great storm or rainfall, an impression leaves an impact on a product which then becomes the formatting foundation of an opinion that can not be altered easily. For this simple reason, creating an interesting, positive, and captivating impression on your customers will go a long way for your business's success and overall reputation.
May 14, 2021
Data, web scraping
Web scraping is becoming a more widely known term. Most associate it with web data extraction, the most efficient and the simplest way of copying large chunks of information online; however, did you know that web scraping was born for a completely different purpose and it took almost two decades for it to transform into web scraping we are familiar with now?
April 30, 2021
Feature, Web Scraper Cloud, Update, release
With constant development process and continuous growth, we are happy to have released a new notification system as well as a data quality feature to allow further automatization optionality.
March 15, 2021
It is no lie that data is power in many ways. For different reasons and applications, different information available online can be used for gaining an advantage in various spheres of life, especially in business.
February 17, 2021
Data, E-commerce, web scraping
Nowadays, with almost everything being available online - many are looking for ways to start a new eCommerce business or automate an existing one.
Imagine a fast-track to improve the efficiency of your new, already existing online store. CSV export from Web Scraper and import to Shopify can get you updating your product lists in only a few minutes.
November 24, 2020
Big Data, Data, Web Scraper Cloud
Data nowadays can be the driving fuel of a company. With the huge increase in technology and data, it has become more important than ever to retrieve, transform, and store that data correctly and effectively.
November 06, 2020
Data, web scraping, Tutorial
Scraping might get hard at times when you're dealing with website structures that frequently change or in general, are hard to scrape with just the point-and-click interface. No need to stress, with the knowledge of the CSS selector - any website structure can be overcome and any website - scraped. Here are the TOP 5 CSS selectors that we frequently use and might be of great use to you.
September 29, 2020
Data, E-commerce, Tutorial
In the previous blogs, we have gone through the classical way of scraping and exposed a tip of how to potentially prevent sitemaps from breaking when an e-commerce site changes product placements in designated categories. Now we are going to take a look at another scraping method which is - scraping with the “Sitemap.xml Links” selector.
September 24, 2020
Data, E-commerce, Tutorial
In the previous blog, we explained how to retrieve the necessary data the classical way by going through the categories, sub-categories, then the products, etc. It is the most primitive and intuitive way of gathering data with Web Scraper extension; however, sometimes a problem arises when the website layout is changed or the placement of various products is altered. For this reason. previously created sitemaps for that specific website may break, or stop working properly.
September 16, 2020
Data, E-commerce, Tutorial
With the enormous growth and development in technology, data being the main driver of modern and fast-growing companies, online business has evolved over the recent years. However, it comes as no surprise since ordering, reserving goods, and services online while not leaving the house is a huge time saver and accessible for everyone with a stable internet connection.
August 27, 2020
Data, Data transformation, Tutorial
With cleaner data, we can begin to transform it. Data transformation can manifest in different forms. It can be clustering, merging, adding information, replacing strings, and so on. OpenRefine covers them all.
August 21, 2020
We are happy to announce that Web Scraper 0.5.0 has been released! This release contains new features such as a new data selection UI engine, a new page load detection system, a welcome page, and a whole lot more!
August 14, 2020
OpenRefine, Data, Data transformation, Tutorial
Data transformation is a step for preparing data sets for AI training and analysis. Clean and transformed data is a vital part of precise and correct data reports and analysis. In this blog series part, we will look at what importance it is to detect and delete blank, inconsistent and duplicate data entries.
July 28, 2020
Data, Data transformation
As the saying goes “garbage in, garbage out”. This can be associated with the idea of data analysis without data transformation beforehand. With bad, messy data, only lousy, chaotic analysis can be done. However, with data transformation, not only the process of analysis becomes easier but also the precision increases.
July 06, 2020
Data, Data post processing, Parser
Have you ever been in a situation when you have two or more data sets of completely different structures? So different that it is impossible to analyze, manage, or integrate. It sounds like every professional’s worst nightmare.
June 05, 2020
If you are a journalist or a data enthusiast and you are in need of data, look no further. We, Web Scraper, offer a collaboration.
May 29, 2020
Data, web scraping
Nowadays, with increasing digitalization, where new technologies are constantly arising and computational power is available almost to everyone, the necessity for data has escalated drastically.
May 12, 2020
Big Data, Data, Global pandemic
Enormous amounts of data that traditional data-processing application software is not capable of dealing with, that is used for analyzing to reveal trends, patterns, associations of especially human behavior and interactions. The subject of Big Data and the usage has increased highly over recent years. With the immense development of technology and the large sets of data that are processed every day, the questions of how to take advantage of it is on everyone’s mind, regardless of the industry.
April 29, 2020
As possibly one of the wisest investments in big data future, data visualization can be a crucial tool for any business. With the continuous advancements in technology and the huge increase in the necessity for big data, visual representation of it can display and showcase many factors of a business that would have been hard to do without data visualization. In this blog, we are gonna explore ways of data representation with the help of a highly easy and versatile tool.
April 08, 2020
Analysis, Google trends
Of everything going on, we took interest in what people are most searching for on google search engine while the coronavirus outbreak.
March 09, 2020
Sports Arbitrage Betting
Imagine a scenario where you invest an amount of money and there is a guaranteed profit! Hard? Impossible? Not at all! Let us provide you with a mathematically proven strategy, which, when applied correctly, actually provides such possibilities.
February 03, 2020
Google Sheets, Query Function
Imagine learning a function, which can work as various other functions; therefore, replacing them with only one and making work with big data easier. With the query function of Google Sheets that scenario is not that absurd because it provides exactly what described. Accessing, summarizing, filtering data from various spreadsheets with only one function - thanks to Google Sheets, the opportunity to do exactly that has been around for quite some time already.
December 23, 2019
Data post processing, Web Scraper Cloud, Parser
We are happy to finally introduce a Parser feature for Web Scraper Cloud.
Usually, to post process data, a custom written script or extra time editing the data manually in a spreadsheet software would be the case; however, the Parser takes care and eases this process.
September 02, 2019
Yellow Pages, Tutorial
Have you ever planned a trip to a different state, city or even a country but did not know any places to have a delicious dinner at? We all know that Yellow Pages is the go-to site if we want to find restaurants in specific areas but it can often be frustrating to get boggled up with bottomless pages on it. Luckily, Web Scraper is here to solve this by allowing you to extract all the information you need from Yellow Pages. So you can enjoy great meals during your trip.
April 17, 2019
We are happy to announce that Web Scraper 0.4.0 has been released. This release contains a new selector, updates to other selectors and improved CSS selector generator. Starting from version 0.4.0 Web Scraper is also available in Firefox.