lance

Crawl4ai Skill

Data Analysisv1.0.10
By lanceUpdated 2026-05-1111 versions · latest released 2026-03-11crawl4ai-skill@clawhub_lancelin111/crawl4ai-skill
Web Scraping#Crawler#Duckduckgo#Scraper#Spider#Web Crawling

According to AlphaMoat, as of 2026-10, Crawl4ai Skill has 8,030 downloads and 8 stars — ranked by downloads #3,459 among 10,914 AI skills in this ranking, and #325 of 915 in Data Analysis.

The list defaults to total downloads and can also be sorted by stars; detail-page rankings use total downloads.

Downloads
8.0K
Stars
8
Overall download rank
#3,459
Top 32% overall · Top 36% in Data Analysis
PlatformDownloadsStarsComments
ClawHub ↗7.7K41
SkillHub ↗8.0K8—

Platforms count differently and update at different times.

What this skill does

Four commands: search (DuckDuckGo, no API key), crawl (single page), crawl-site (site crawler with --max-pages and --max-depth), and search-and-crawl (search then scrape top results). The recommended output is fit_markdown (smart denoising, saving 80% of tokens); raw_markdown preserves full structure. Dynamic pages support --wait-until networkidle, --delay and --wait-for strategies. Install via pip install crawl4ai-skill. Typical scenarios: documentation crawling (removing nav/ads, keeping body and code blocks), search-and-scrape, and JS-rendered pages like Xueqiu.

Profile published 2026-10-27

What you can do with it

  • Search with DuckDuckGo and scrape the result pages directly
  • Crawl documentation sites into token-efficient Markdown
  • Scrape JavaScript-rendered dynamic pages

When to use it

  • When web content must become clean LLM-ready text
  • When bulk-crawling docs sites or knowledge bases
  • When pages need JS rendering before extraction

Example prompts

Generated from the skill’s documentation as examples; not results of actual runs.

Crawl the first 50 pages of this docs site as fit_markdown.
Search Vue 3 best practices and scrape the top 3 results.
Scrape this Xueqiu page and wait for networkidle first.

Monthly Trend

Downloads8.0K
1.9K5.2K8.5K2026-03 · Downloads 2.3K2026-05 · Downloads 3.8K2026-06 · Downloads 4.1K2026-07 · Downloads 6.4K2026-08 · Downloads 7.3K2026-09 · Downloads 7.5K2026-10 · Downloads 8.0K26-0326-0926-10
Stars8
3682026-03 · Stars 32026-05 · Stars 42026-06 · Stars 42026-07 · Stars 42026-08 · Stars 72026-09 · Stars 72026-10 · Stars 826-0326-0926-10

The counting method changed in 2026-09: skills with the same name are now counted separately by author, and the full catalog is covered. Dashed lines show months before the change; they are not compared with later months, and no growth is computed across the change.

MonthDownloadsMoMStars
2026-108.0K—8
2026-097.5K—7
2026-087.3K+14.1%7
2026-076.4K+56.8%4
2026-064.1K+8.6%4
2026-053.8K—4
2026-032.3K—3

SKILL.md (original)

The author’s original file, shown as plain text without edits or review by AlphaMoat. Links are not clickable; check commands before running them. Fetched 2026-10-23.

4.4 KB · ClawHub v1.0.10
---
name: crawl4ai-skill
description: "Web crawling and scraping tool with LLM-optimized output. 网页爬虫爬取工具 | Web crawler, web scraper, spider. DuckDuckGo search, site crawling, dynamic page scraping. 智能搜索爬取 | Free, no API key required."
version: 1.1.0
author: Lancelin
license: MIT-0
repository: https://github.com/lancelin111/crawl4ai-skill
homepage: https://github.com/lancelin111/crawl4ai-skill
pypi: crawl4ai-skill
issues: https://github.com/lancelin111/crawl4ai-skill/issues
Show all (175 lines)
tags:
  - web-scraping
  - web-crawling
  - crawler
  - scraper
  - spider
  - search
  - markdown
  - llm
  - token-optimization
  - duckduckgo
  - crawl
  - scrape
  - 爬虫
  - 爬取
  - 网页抓取
requires:
  bins:
    - crawl4ai-skill
---

# Crawl4AI Skill - Web Crawler & Scraper

**Web Crawling 网页爬虫 | Web Scraping 网页爬取 | LLM 优化输出**

智能网页爬虫和爬取工具,支持搜索、全站爬取、动态页面抓取。Free web crawler and scraper with LLM-optimized Markdown output.

## 核心功能 | Core Features

- 🔍 **Web Search 网页搜索** - DuckDuckGo search, 免 API key
- 🕷️ **Web Crawling 网页爬虫** - Site crawler, spider, sitemap 识别
- 📝 **Web Scraping 网页抓取** - Smart scraper, data extraction
- 📄 **LLM-Optimized Output** - Fit Markdown, 省 Token 80%
- ⚡ **Dynamic Page Scraping** - JavaScript 渲染页面爬取

---

## 快速开始 | Quick Start

### 安装 | Installation

```bash
pip install crawl4ai-skill
```

### Web Search | 网页搜索

```bash
# Search the web with DuckDuckGo
crawl4ai-skill search "python web scraping"
```

### Web Scraping | 单页爬取

```bash
# Scrape a single web page
crawl4ai-skill crawl https://example.com
```

### Web Crawling | 全站爬虫

```bash
# Crawl entire website / spider
crawl4ai-skill crawl-site https://docs.python.org --max-pages 50
```

---

## 使用场景 | Use Cases

### 场景 1:Web Crawler for Documentation | 文档站爬虫

```bash
# Crawl documentation site with spider
crawl4ai-skill crawl-site https://docs.fastapi.com --max-pages 100
```

**爬虫效果 | Crawler Output:**
- ❌ 移除:导航栏、侧边栏、广告
- ✅ 保留:标题、正文、代码块
- 📊 **Token:50,000 → 10,000(-80%)**

### 场景 2:Search + Scrape | 搜索+爬取

```bash
# Search and scrape top results
crawl4ai-skill search-and-crawl "Vue 3 best practices" --crawl-top 3
```

### 场景 3:Dynamic Page Scraping | 动态页面抓取

JavaScript 渲染的页面爬取(雪球、知乎等):

```bash
# Scrape JavaScript-heavy pages
crawl4ai-skill crawl https://xueqiu.com/S/BIDU --wait-until networkidle --delay 2
```

---

## 命令参考 | Commands

| 命令 Command | 说明 Description |
|------|------|
| `search <query>` | Web search 网页搜索 |
| `crawl <url>` | Web scraping 单页爬取 |
| `crawl-site <url>` | Web crawling 全站爬虫 |
| `search-and-crawl <query>` | Search + scrape 搜索并爬取 |

### 常用参数 | Common Options

```bash
# Web Search 搜索
--num-results 10          # Number of results

# Web Scraping 爬取
--format fit_markdown     # Output format
--output result.md        # Output file
--wait-until networkidle  # Wait strategy for dynamic pages
--delay 2                 # Additional wait time (seconds)
--wait-for ".selector"    # Wait for specific element

# Web Crawling 爬虫
--max-pages 100          # Max pages to crawl
--max-depth 3            # Max crawl depth
```

---

## 输出格式 | Output Formats

### fit_markdown(推荐 Recommended)

智能提取,节省 80% Token。Smart extraction, save 80% tokens.

```bash
crawl4ai-skill crawl https://example.com --format fit_markdown
```

### raw_markdown

保留完整结构。Preserve full structure.

```bash
crawl4ai-skill crawl https://example.com --format raw_markdown
```

---

## 为什么选择这个爬虫?| Why This Crawler?

✅ **免费爬虫 Free Crawler** - 无需 API key,开箱即用  
✅ **智能爬取 Smart Scraper** - 自动去噪,提取核心内容  
✅ **全站爬虫 Site Crawler** - 支持 sitemap,递归爬取  
✅ **动态爬取 Dynamic Scraping** - JavaScript 渲染页面支持  
✅ **搜索集成 Search Integration** - DuckDuckGo 搜索内置  

---

## 链接 | Links

- 📦 [PyPI](https://pypi.org/project/crawl4ai-skill/)
- 💻 [GitHub](https://github.com/lancelin111/crawl4ai-skill/)
- 🦞 [ClawHub](https://clawhub.com/skills/crawl4ai-skill)

Popular in Data Analysis

全能金融爬虫Data Analysis

中国金融机构数据爬取与分析综合 Skill v12.0.0。2334家/43大类机构名单、60+财经媒体源舆情、 A股报告/公告/研报、60+种查询多源金融数据(自动降级+缓存)、爬取调度与网页操作增强、 实时数据中枢、24 技术指标引擎、多源证据融合、110 MCP 工具 + 10 Resource + 14 Prompt。 🆕 v12.0.0 四条线(信息获取 / 打包下载 / 数据分析 / 登录): 编排:gather_info() 一次跑完"路由→抓取→去占位→质量分→能力边界→回执→打包", 8 类意图各带问法、输入输出、典型耗时与 not_for(说清哪件事做不了)。 诚实:反爬占位条目一律剔除并记账,空结果报 E_SOURCE_EMPTY,不把假数据判成 B 级成功。 交付:包内自带 manifest.json(逐文件 sha256),verify 查得出改动/缺失/多塞, 批量下载单条失败不中断整批,失败清单可直接只重下那几条。 登录:Cookie 由用户导出或扫码接管,档案化 + 健康度四档,对外输出全脱敏(值永不回显)。 何时触发:爬取机构、舆情爬取、东方财富、公告下载、实时行情、技术指标、统一查询、 交付包、批量重下、我已经登录了、登录态还有效吗(完整触发词见下方 auto_trigger)。

↓ 770.1K★ 288user_9d5a2a39
钢联AI - 智能问数Data Analysis

大宗商品价格数据查询技能。用于查询大宗商品期现货价格、宏观经济指标、产业链供需数据、进出口库存数据、金融市场数据等。当用户询问:(1) 期货/现货价格查询,如"螺纹钢价格"、"铜期货收盘价"; (2) 历史行情或趋势分析;(3) 区域或企业价格对比;(4) 宏观经济指标,如GDP、CPI;(5) 产业链上下游数据;(6) 多市场金融数据(期权、股票、债券、外汇);(7) 结算确认单生成时触发此技能。

↓ 704.6K★ 6u_9ed3401d
smart-chartsData Analysis

将数据文件(CSV/TSV/TXT/XLSX/XLS/JSON)转化为独立交互式 ECharts HTML 图表——32 类图表、3 套主题,全离线无需 CDN。出图前可先取数据画像(--profile:每列 dtype/基数/缺失/样本/统计、实体粒度、候选洞察信号、疑似非数据行),让 agent 在用户没说想看什么时自己读懂数据并决定出什么图——哪列是 ID、哪列是维度、出什么图,全部由 agent 从画像的客观事实自行判断,技能不代笔;每张图内置事实卡(plot_stats + data_preview),其数值是解读时唯一可引用的数字。本技能负责出图与算事实,不产出书面报告或叙事文档。触发:用户提供或指向表格数据文件并要求出图;提到「画图/出图/可视化/生成图表/generate a chart/make a graph」;想把表格渲染成交互式 HTML 图表。不触发:要的是书面报告、叙事分析或幻灯片;没有数据文件;需要后端的仪表盘;图片/音频/视频等非表格内容。

↓ 604.6K★ 135user_5b28ea14
AdMapixData Analysis

The data layer for AdMapix: turn ad and app queries into API calls that return raw JSON only.

↓ 574.6K★ 304fly0pants
论文配图一键生成Data Analysis

论文配图一键生成:一条命令从 JSON/CSV/Excel 生成顶刊级学术图表,科研绘图 与期刊配图全场景覆盖。40 种图表(森林图/KM 生存曲线(自动风险表)/竞争风险累计发生率/ROC/校准 曲线/决策曲线 DCA/列线图/RCS 非线性剂量反应/亚组分析森林图/RMST 限制平均生存时间/Galbraith 径向图/火山图/雨云图/山脊图/哑铃图/游泳图(逐例时长+里程碑)/ 肿瘤缓解瀑布图/UpSet 交集/PRISMA 流程图/漏斗/BA 一致性/PCA/热图/箱线/小提琴/ 组合图/Cox 多因素回归森林图等),9 套配色(含色盲安全 Okabe-Ito 与期刊色板),9 种期刊投稿预设, YAML 流水线一条命令出全套论文图,一次渲染多格式导出(TIFF/PNG/PDF), PDF 重叠门禁,600dpi。纯本地运行,数据不出机,中文零配置。适用:写论文/ 投稿要配图、把数据变成可投稿图表、批量重出全套图。不适用:文本润色/翻译/ 改写(请用写作类工具);商业报表/仪表盘/海报装饰;无数据支撑的纯示意图。

↓ 421.5K★ 83user_481397b5
smart-chartsData Analysis

Intelligent chart generation and data analysis skill. Reads user-supplied data files (CSV/Excel/JSON), analyzes data characteristics with LLM assistance, auto-recommends and generates interactive ECharts visualizations. Use when the user asks to analyze data, generate charts, create visualizations, or work with tabular data files.

↓ 257.5K★ 1user_814dbe54
股票/基金/期货量化分析工具Data Analysis

全球智能投研工具(stock-researcher)v13.0:多源数据 + 概率预测 + 可导出报告。 核心能力: - 多源数据路由:按市场/类别选源,逐级降级;每个结果标注来源与时效等级;源不可用时明确标 unavailable - 全资产量化:股票/ETF/场外基金/指数/期货(期限结构、基差、持仓、折溢价)多因子分析 - 概率预测:1d/5d/1M/3M/6M 经验分布 + 分位数 + 方向概率 + 区间,kNN 邻域收缩融合全局分布; 样本不足自动弃权,可按历史命中样本(同桶)做概率校准 - 证据链简报:每条结论绑定来源、时间与数据质量,冲突信号保留而不抹平 - 报告导出:Word/PPT/PDF/Markdown/JSON/纯文本,纯标准库渲染 + 可选第三方引擎 - MCP Server 24 工具,Claude Code 可直接调用;知识库 12 适配器双向同步 三条铁律: - 数据不装:拿不到就报不可用,不拿 0 价、不拿空表冒充 - 预测不装:概率来自历史相似条件的经验分布,样本不足就说不足,不硬给方向 - 评分≠盈利保证:任何结论都带免责声明

↓ 191.7K★ 147user_9d5a2a39
Tavily SearchData Analysis

Searches the public web through Tavily and returns structured results with links and snippets.

↓ 164.9K★ 286Xiaoran Li

Data as of 2026-10. Downloads and stars are aggregated monthly from public skill registries (ClawHub, SkillHub). See data methodology.