By bytesagain3Updated 2026-05-1110 versions · latest released 2026-03-23crawler@clawhub_bytesagain3/crawler
Web Scraping#Crawler#Robots Txt#Scrapy#Selenium#Web Scraping
According to AlphaMoat, as of 2026-09, Crawler has 2,289 downloads and 0 stars — ranked #4,240 of 5,078 AI skills overall, and #322 of 399 in Data Analysis.
| Platform | Installs | Downloads | Stars | Comments |
|---|---|---|---|---|
| ClawHub ↗ | 63 | 2.3K | - | - |
| SkillHub ↗ | 122 | 2.3K | - | — |
Platforms count differently and update at different times.
It provides plain-text reference material without calling external APIs, requiring credentials, or accessing the network itself. Apply the guidance only to lawful, permissioned crawling; prefer official APIs, follow site terms and robots.txt, and avoid bypassing access controls with proxies, fingerprints, or CAPTCHA tactics.
Profile published 2026-10-29
Generated from the skill’s documentation as examples; not results of actual runs.
The counting method changed in 2026-09: skills with the same name are now counted separately by author, and the full catalog is covered. Dashed lines show months before the change; they are not compared with later months, and no growth is computed across the change.
| Month | Downloads | MoM | Stars |
|---|---|---|---|
| 2026-09 | 2.3K | — | - |
| 2026-08 | 2.1K | +12.0% | - |
| 2026-07 | 1.9K | +19.4% | - |
| 2026-06 | 1.6K | +12.0% | - |
| 2026-05 | 1.4K | — | - |
| 2026-03 | 711 | — | - |
The author’s original file, shown as plain text without edits or review by AlphaMoat. Links are not clickable; check commands before running them. Fetched 2026-10-23.
--- name: "crawler" version: "3.0.0" description: "Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations" author: "BytesAgain" homepage: "https://bytesagain.com" source: "https://github.com/bytesagain/ai-skills" tags: [web-scraping, scrapy, crawler, robots-txt, selenium] category: "devtools" ---
# Crawler Web crawling and scraping reference — robots.txt protocol, Scrapy framework, anti-bot detection, headless browsers, and legal considerations. No API keys or credentials required — outputs reference documentation only. ## Commands | Command | Description | |---------|-------------| | `intro` | Crawling vs scraping, robots.txt, sitemap | | `standards` | HTTP caching, structured data, meta tags | | `troubleshooting` | Anti-bot detection, JS rendering, encoding | | `performance` | Concurrency, dedup, incremental, distributed | | `security` | Legal landscape, ethical guidelines, proxies | | `migration` | BeautifulSoup to Scrapy, requests to Playwright | | `cheatsheet` | Scrapy commands, CSS/XPath, curl, user-agents | | `faq` | Legality, JS pages, blocking, storage | ## Output Format All commands output plain-text reference documentation via heredoc. No external API calls, no credentials needed, no network access. --- *Powered by BytesAgain | bytesagain.com | [email protected]*
The data layer for AdMapix: turn ad and app queries into API calls that return raw JSON only.
Searches the public web through Tavily and returns structured results with links and snippets.
A data-analysis skill that turns databases, spreadsheets, and raw exports into decision-ready conclusions.
A skill that analyzes stocks and cryptocurrencies on Yahoo Finance data and tracks a portfolio.
Manages a local stock watchlist and summarizes recent quotes for its codes from the Tonghuashun stock pages.
知了标讯: Searches and analyzes tender, award, supplier, and procurement-market information.
Track blog and RSS/Atom feed updates from the command line.
Summarises per-model usage cost for Codex or Claude.
Data month: 2026-09 · Downloads, stars and installs are aggregated monthly from public skill registries (ClawHub, SkillHub). See methodology.