How do I scrap data over Python

Difference Between BeautifulSoup and Scrapy Crawler?

I want to create a website that shows the comparison between Amazon and E-Bay product price. Which one works better and why? I am a bit familiar with BeautifulSoup, but not so much with Scrapy Crawler .


Scrapy is a web spider or Web scraper framework . You give Scrapy a root url to start crawling. Then you can set restrictions on the number (number) of URLs that you want to crawl and get. It is a complete framework for web scraping or Crawling .


BeautifulSoup is a Analysis library, which is also very good at pulling content from URLs and allowing you to easily analyze certain parts of them. It just gets the contents of the url you specified and then stops. It won't be crawled unless you manually put it into an infinite loop with certain criteria.

In simple terms, Beautiful Soup lets you build something similar to Scrapy. Beautiful Soup is one Library, during scrapy on complete framework is .


I think both are good ... I'm currently doing a project that uses both. First, I scrap all the pages with Scrapy and save them in a Mongodb collection with their pipelines. Also, I download the images that are on the page. After that, I use BeautifulSoup4 to do some pos processing where I need to change attribute values ​​and get some special tags.

If you don't know which page products you want, a good tool is difficult because its crawlers allow you to run all the Amazon / eBay websites that are looking for the products without creating an explicit for loop.

Check out the Scrapy documentation, it's very easy to use.

Both use to parse data.

Scrapy :

  • Scrapy is a fast, high-level crawling and web scraping framework that crawls websites and extracts structured data from their pages.
  • However, there are some limitations when data comes from Java scripts or is loaded dynamically. We can overcome them with the help of packages like Splash, Selenium, etc.

BeautifulSoup :

  • Beautiful Soup is a Python library for retrieving data from HTML and XML files.

  • We can use this package to pull data from Java scripts or load pages dynamically.

Scrapy with BeautifulSoup is one of the best combinations that we can use to scrape off static and dynamic content

The way I do this is to use the eBay / Amazon APIs instead of Scrapy and then analyze the results with BeautifulSoup.

The APIs give you an official way to get the same data you would have gotten from Scrapy Crawler without having to worry about hiding your identity, messing around with proxies, etc.

Scrapy It's about a Web scraping framework, that has tons of extras that make scraping easy so we can just focus on crawling logic. Some of my favorite things Scrapy does for us are listed below.

  • Feed exports: Basically we can save data in different formats like CSV, JSON, jsonlines and XML.
  • Asynchronous scraping: Scrapy uses a twisted framework that allows us to call multiple URLs at once, with non-blocking processing of each request (basically we don't have to wait for a request to complete before sending another request).
  • Selectors: Here we can compare scrapy to beautiful soup. Selectors allow us to select certain data from the website, such as B. Headings, certain divs with a class name, etc.). Scrapy uses lxml to parse, which is extremely fast as a nice soup.
  • Setting proxy, user agent, headers etc .: With Scrapy we can set and rotate proxy and other headers dynamically.

  • Item Pipelines: With pipelines we can process data after extraction. For example, we can configure the pipeline to transfer data to your MySQL server.

  • Cookies: Scrapy automatically processes cookies for us.


TLDR: Scrapy is a framework that provides everything you need to create large-scale crawls. It offers various features that hide the complexity of crawling the websites. One can just start writing web crawlers without worrying about the setup burden.

Nice soup Nice soup is a python package for Parsing of HTML and XML documents . So with Beautiful Soup you can analyze a webpage that has already been downloaded. BS4 is very popular and old. Unlike scrapy, you can You don't just use beautiful soups to make crawlers . You need other libraries like queries, urllib, etc. to crawl with bs4. This in turn means that you have to manage the list of URLs crawled, crawled, process cookies, manage proxies, handle errors, create your own functions to send data to CSV, JSON, XML, etc. If you want to speed it up then you have to use other libraries like multiprocessing.

To conclude.

  • Scrapy is a comprehensive framework that allows you to write crawlers without any problems.

  • Nice Soup is a library that you can use to analyze a web page. It cannot be used to scrape web on its own.

You should definitely be using Scrapy for your Amazon and E-Bay product price comparison website. You can create a database of urls and run the crawler every day (cron jobs, celery to schedule crawls) and update the price for your database. That way, your website is always pulled from the database, and the crawler and database act as separate components.

BeautifulSoup is a library that allows you to extract information from a web page.

Scrapy however, is a framework that does the above and many more Performing tasks that you will likely need in your scraping project, e.g. B. Pipelines for storing data.

You can check out this blog to get started with scrapy. Https://

With Can scrapy You save tons of code and start structured programming. If you don't like any of the pre-written methods of Scapy, you can BeautifulSoup can be used in place of the scrapy method. A large project has both advantages.

The differences are many and the selection of a tool / technology depends on individual needs.

Some key differences are:

  1. BeautifulSoup is comparative easy to learn as scrapy.
  2. The extensions, support, and community are bigger for Scrapy than they are for BeautifulSoup.
  3. Scrapy should be used as a Be considered spider, while BeautifulSoup one Parser is .

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from.

By continuing, you consent to our use of cookies and other tracking technologies and affirm you're at least 16 years old or have consent from a parent or guardian.

You can read details in our Cookie policy and Privacy policy.