Awesome-crawler

A collection of awesome web crawler,spider and resources in different language

Python

Scrapy - A fast high-level screen scraping and web crawling framework.
- django-dynamic-scraper - Creating Scrapy scrapers via the Django admin interface
pyspider - A powerful spider system.
cola - A distributed crawling framework.
Demiurge - PyQuery-based scraping micro-framework.
Scrapely - A pure-python HTML screen-scraping library.
feedparser - Universal feed parser.
you-get - Dumb downloader that scrapes the web.
Grab - Site scraping framework.
MechanicalSoup - A Python library for automating interaction with websites.
portia - Visual scraping for Scrapy.
crawley - Pythonic Crawling / Scraping Framework based on Non Blocking I/O operations.
RoboBrowser - A simple, Pythonic library for browsing the web without a standalone web browser.
MSpider - A simple ,easy spider using gevent and js render.
brownant - A lightweight web data extracting framework.

Apache Nutch - Highly extensible, highly scalable web crawler for production environment.
Crawler4j - Simple and lightweight web crawler.
JSoup - Scrapes, parses, manipulates and cleans HTML.
websphinx - Website-Specific Processors for HTML INformation eXtraction.
Open Search Server - A full set of search functions. Build your own indexing strategy. Parsers extract full-text data. The crawlers can index everything.
Gecco - A easy to use lightweight web crawler
WebCollector - Simple interfaces for crawling the Web,you can setup a multi-threaded web crawler in less than 5 minutes.
Webmagic - A scalable crawler framework.
Heritrix3 - Extensible, web-scale, archival-quality web crawler project.
SeimiCrawler - An agile, distributed crawler framework.

ccrawler - Built in C# 3.5 version. it contains a simple extention of web content categorizer, which can saparate between the web page depending on their content.
SimpleCrawler - Simple spider base on mutithreading, regluar expression.
DotnetSpider - This is a cross platfrom, ligth spider develop by C#.
Abot - C# web crawler built for speed and flexibility.
Hawk - Advanced Crawler and ETL tool written in C#/WPF.
SkyScraper - An asynchronous web scraper / web crawler using async / await and Reactive Extensions.

Goutte - A screen scraping and web crawling library for PHP.
- laravel-goutte - Laravel 5 Facade for Goutte.
dom-crawler - The DomCrawler component eases DOM navigation for HTML and XML documents.
pspider - Parallel web crawler written in PHP.
php-spider - A configurable and extensible PHP web spider.

open-source-search-engine - A distributed open source search engine and spider/crawler written in C/C++.

upton - A batteries-included framework for easy web-scraping. Just add CSS(Or do more).
wombat - Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.
RubyRetriever - RubyRetriever is a Web Crawler, Scraper & File Harvester.
Spidr - Spider a site ,multiple domains, certain links or infinitely.
Cobweb - Web crawler with very flexible crawling options, standalone or using sidekiq.

pholcus - A distributed, high concurrency and powerful web crawler.
gocrawl - Polite, slim and concurrent web crawler.
fetchbot - A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
go_spider - An awesome Go concurrent Crawler(spider) framework.
dht - BitTorrent DHT Protocol && DHT Spider.
ants-go - A open source, distributed, restful crawler engine in golang.
scrape - A simple, higher level interface for Go web scraping.

crawler - Scala DSL for web crawling.
scrala - Scala crawler(spider) framework, inspired by scrapy.
ferrit - Ferrit is a web crawler service written in Scala using Akka, Spray and Cassandra.