1. Install scrapy -- pip install scrapy
  2. Create a project, go to your directory of choice using cd "your directory" 
  3. Run scrapy startproject NamOfProject
  4. Using custom proxies with your Project  - head to settings.py
  5. Uncomment DOWNLOADER_MIDDLEWARES
  6. Inside DOWNLOADER_MIDDLEWARES put the followings:
DOWNLOADER_MIDDLEWARES = {
   'postscrape.middlewares.PostscrapeDownloaderMiddleware': 543,
   'postscrape.middlewares.ProxyMiddleware': 1,
}

 

head to the middlewares.py

create the class you specified in the DOWNLOADER_MIDDLEWARES as shown below:

class ProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = "httpproxy.yourproxy.name"

 

Test your crawler:

scrapy shell https://blog.scrapinghub.com/

Creating a spider inside the File.

We want to create a spider class inside spiders folder,  to do this we can create a new file inside Spiders and give it a name, for example:

import scrapy


class BBCSpider(scrapy.Spider):
    name = 'bbcSpider'
    start_urls = [
        'https://www.bbc.com/news'
    ]

    def parse(self, response):
        
        for post in response.css('.nw-c-top-stories__secondary-item'):
            yield {
                'header' : post.css('.gs-c-promo-heading h3::text').get(),
                'summary' : post.css('.gs-c-promo-summary::text').get(),
                'top_news' : response.css('.nw-c-top-stories__primary-item h3::text').get()

           }

the name you give it at this stage will be used in spider when you call the spider to crawl, not the name of the class but the name you give as an argument inside the class.

start_url, will be the url your scrapy use to collect html.

parse method, is the method used to parse the html data for example in the parse method above, we loop through a specific scc class and get the header, summary and top news 

to run this we can call 

scrapy list -- this will show all of our spider classes if there are any issues with the class it will not show up here.

finally to actually run the file, we can use:

scrapy crawl bbcSpider

 

 

Now let's see how the full code looks like which would crawl bbc, and save the output in the location of choice in your pc via pandas data frame

settings.py

# Scrapy settings for bbc project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

BOT_NAME = 'bbc'

SPIDER_MODULES = ['bbc.spiders']
NEWSPIDER_MODULE = 'bbc.spiders'


# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'bbc (+http://www.yourdomain.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#   'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#   'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#    'bbc.middlewares.BbcSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
   'bbc.middlewares.BbcDownloaderMiddleware': 543,
   'bbc.middlewares.ProxyMiddleware': 1,
}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#    'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
#ITEM_PIPELINES = {
#    'bbc.pipelines.BbcPipeline': 300,
#}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

 

 

middlewares.py

# Define here the models for your spider middleware
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/spider-middleware.html

from scrapy import signals
import scrapy


class ProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = "httpproxy.yourproxy.ofchoice"

class BbcSpiderMiddleware(object):
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the spider middleware does not modify the
    # passed objects.

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_spider_input(self, response, spider):
        # Called for each response that goes through the spider
        # middleware and into the spider.

        # Should return None or raise an exception.
        return None

    def process_spider_output(self, response, result, spider):
        # Called with the results returned from the Spider, after
        # it has processed the response.

        # Must return an iterable of Request, dict or Item objects.
        for i in result:
            yield i

    def process_spider_exception(self, response, exception, spider):
        # Called when a spider or process_spider_input() method
        # (from other spider middleware) raises an exception.

        # Should return either None or an iterable of Request, dict
        # or Item objects.
        pass

    def process_start_requests(self, start_requests, spider):
        # Called with the start requests of the spider, and works
        # similarly to the process_spider_output() method, except
        # that it doesn’t have a response associated.

        # Must return only requests (not items).
        for r in start_requests:
            yield r

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)


class BbcDownloaderMiddleware(object):
    # Not all methods need to be defined. If a method is not defined,
    # scrapy acts as if the downloader middleware does not modify the
    # passed objects.

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_request(self, request, spider):
        # Called for each request that goes through the downloader
        # middleware.

        # Must either:
        # - return None: continue processing this request
        # - or return a Response object
        # - or return a Request object
        # - or raise IgnoreRequest: process_exception() methods of
        #   installed downloader middleware will be called
        return None

    def process_response(self, request, response, spider):
        # Called with the response returned from the downloader.

        # Must either;
        # - return a Response object
        # - return a Request object
        # - or raise IgnoreRequest
        return response

    def process_exception(self, request, exception, spider):
        # Called when a download handler or a process_request()
        # (from other downloader middleware) raises an exception.

        # Must either:
        # - return None: continue processing this exception
        # - return a Response object: stops process_exception() chain
        # - return a Request object: stops process_exception() chain
        pass

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)

 

 

bbc_spider.py

import scrapy
import pandas as pd

output_location = r"your\file\location"

class BBCSpider(scrapy.Spider):
    name = 'bbcSpider'
    start_urls = [
        'https://www.bbc.com/news'
    ]

    def parse(self, response):
        items = []
        for post in response.css('.nw-c-top-stories__secondary-item'):
            item =  {
                'header' : post.css('.gs-c-promo-heading h3::text').get(),
                'summary' : post.css('.gs-c-promo-summary::text').get(),
                'top_news' : response.css('.nw-c-top-stories__primary-item h3::text').get(),
                'section' :post.css('.gs-c-section-link span::text').get(),
                'time_of_post': post.css('.gs-c-timestamp time::attr(datetime)').get()

            }
            items.append(item)

            df = pd.DataFrame(items, columns = ['header', 'summary', 'top_news','section', 'time_of_post'])
            df.to_csv(output_location + "top_news.csv")