Web crawler and scraper for Rust
-
Updated
Mar 25, 2026 - Rust
#
Crawler
A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
Here are 119 public repositories matching this topic...
An ergonomic Python HTTP Client with TLS fingerprint
python tls rust http crawler scraper https websocket http2 http-client fingerprint websocket-client web-scraping akamai ja3 tls-client tls-fingerprint ja4 akamai-fingerprint
-
Updated
Mar 14, 2026 - Rust
An ergonomic Rust HTTP Client with TLS fingerprint
tls rust http client crawler scraper https websocket http2 http-client fingerprint web-scraper websocket-client akamai ja3 tls-fingerprint ja4
-
Updated
Mar 25, 2026 - Rust
SiteOne Crawler is a cross-platform website crawler and analyzer for SEO, security, accessibility, and performance optimization—ideal for developers, DevOps, QA engineers, and consultants. Supports Windows, macOS, and Linux (x64 and arm64).
testing security website qa crawler performance seo crawling analyzer stress-testing swoole seotools quality-assessment
-
Updated
Mar 17, 2026 - Rust
A rock-solid cryptocurrency crawler library.
-
Updated
Mar 19, 2023 - Rust
🕷️ The pipeline for the OSCAR corpus
-
Updated
Nov 9, 2025 - Rust
Dyer is designed for reliable, flexible and fast web crawling, providing some high-level, comprehensive features without compromising speed.
-
Updated
Aug 5, 2025 - Rust
✴️ An experimental graph database
-
Updated
Jul 20, 2023 - Rust
Spider ported to Python
-
Updated
Jan 20, 2026 - Rust
🎓a Better BlackBoard for PKUers. 北京大学教学网命令行工具（🖥️Win/🐧Linux/🍏Mac）, 支持查看/提交作业、下载课程回放.
-
Updated
Mar 12, 2026 - Rust
A command line tool based on the crypto-crawler library.
-
Updated
Mar 19, 2023 - Rust
Spider ported to Node.js
-
Updated
Jan 25, 2026 - Rust
CLI to download all images/webms in a 4chan thread
-
Updated
Aug 22, 2022 - Rust
Kabegame — An anime image crawler client with pluggable crawlers (from a GitHub plugin repo), wallpaper rotation by custom rules, and Wallpaper Engine export. Supports Windows 10/11, macOS Big Sur+, and Ubuntu.
-
Updated
Mar 25, 2026 - Rust
⚡ A subdomain enumeration tool leveraging diverse techniques, designed for advanced pentesting operations
rust crawler scanner bruteforce rust-lang subdomain brute-force pentesting pentest rust-crate pentest-tool subdomain-scanner subdomain-enumeration subdomain-bruteforcing zonetransfer pentesting-tool subdomain-finder searchengines
-
Updated
Mar 23, 2026 - Rust
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
markdown rust cli crawler scraper ai mcp html-to-markdown web-crawler web-scraper self-hosted web-scraping data-extraction webscraping ai-agents web-extraction llm ai-scraping tls-fingerprinting mcp-server
-
Updated
Mar 24, 2026 - Rust
Rust Web Crawler saving pages on Redis
-
Updated
Jun 13, 2021 - Rust
Crawling and scraping the Web for fun and profit
-
Updated
Apr 13, 2023 - Rust
- Followers
- 550 followers
- Website
- github.com/topics/crawler
- Wikipedia
- Wikipedia