AlphaMoatによると、2026-10時点でScraplingのダウンロード数は10,935、スター数は9で、累計ダウンロード数で今回のランキング対象AIスキル10,914件中2,502位、データ分析では915件中234位です。
一覧の初期表示は累計ダウンロード数順で、スター数順にも並べ替えできます。詳細ページの順位は累計ダウンロード数で計算しています。
| プラットフォーム | ダウンロード数 | スター数 | コメント |
|---|---|---|---|
| ClawHub ↗ | 8295 | 6 | - |
| SkillHub ↗ | 1.1万 | 9 | — |
プラットフォームごとに集計方法と集計時点が異なります。
取得と解析に加え、ページ構造が変わっても類似度で目的の要素を拾い直す適応型のセレクタを備え、よくある bot 回避を越える選択肢やサイトマップからの巡回も用意します。導入は pip の一行で、実行には Python が要ります。ログインが必要なサイトは自前の認証情報が要り、許された公開サイトだけを対象にすべきだと原本は断っています。 使用前に送信先/アカウントとデータ共有を確認してください。重要な外部操作は実行前に確認し、生成・取得結果を独立して検証します。
資料公開日 2026-10-27
スキルのドキュメントをもとに生成した例であり、実際の実行結果ではありません。
2026-09 から集計方法を変更し、同名スキルを作者ごとに分けて集計し、全スキルを対象にしました。破線は変更前の月で、変更後のデータとは直接比較せず、変更をまたぐ増減も計算していません。 同名の別スキルがあるため、変更前のデータは現在ダウンロード数が最も多いこのスキルに帰属させています。
| 月 | ダウンロード数 | 前月比 | スター数 |
|---|---|---|---|
| 2026-10 | 1.1万 | — | 9 |
| 2026-09 | 8113 | — | 9 |
| 2026-08 | 7658 | +5.8% | 8 |
| 2026-07 | 7240 | +12.7% | 8 |
| 2026-06 | 6422 | +10.0% | 7 |
| 2026-05 | 5838 | — | 5 |
| 2026-03 | 2776 | — | 3 |
作者が公開した元のファイルをテキストのまま表示しています。AlphaMoat による編集や審査はしていません。リンクはクリックできません。コマンドは内容を確認してから実行してください。取得日 2026-10-23。
---
name: scrapling
description: "Adaptive web scraping framework with anti-bot bypass and spider crawling."
version: "1.0.8"
metadata:
{"openclaw":{"emoji":"🕷️","requires":{"bins":["python3"]}, "tags":["web-scraping", "crawling", "research", "automation"]}}
---
# Scrapling - Adaptive Web Scraping
> "Effortless web scraping for the modern web."
---
## Credits
### Core Library
- **Repository:** https://github.com/D4Vinci/Scrapling
- **Author:** D4Vinci (Karim Shoair)
- **License:** BSD-3-Clause
- **Documentation:** https://scrapling.readthedocs.io
### API Reverse Engineering Methodology
- **GitHub:** https://github.com/paoloanzn/free-solscan-api
- **X Post:** https://x.com/paoloanzn/status/2026361234032046319
- **Author:** @paoloanzn
- **Insight:** "Web scraping is 80% reverse engineering"
---
## Installation
```bash
# Core library (parser only)
pip install scrapling
# With fetchers (HTTP + browser automation) - RECOMMENDED
pip install "scrapling[fetchers]"
scrapling install
# With shell (CLI tools) - RECOMMENDED
pip install "scrapling[shell]"
# With AI (MCP server) - OPTIONAL
pip install "scrapling[ai]"
# Everything
pip install "scrapling[all]"
# Browser for stealth/dynamic mode
playwright install chromium
# For Cloudflare bypass (advanced)
pip install cloudscraper
```
---
## Agent Instructions
### When to Use Scrapling
**Use Scrapling when:**
- Research topics from websites
- Extract data from blogs, news sites, docs
- Crawl multiple pages with Spider
- Gather content for summaries
- Extract brand data from any website
- Reverse engineer APIs from websites
**Do NOT use for:**
- X/Twitter (use x-tweet-fetcher skill)
- Login-protected sites (unless credentials provided)
- Paywalled content (respect robots.txt)
- Sites that prohibit scraping in their TOS
---
## Quick Commands
### 1. Basic Fetch (Most Common)
```python
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://example.com')
# Extract content
title = page.css('h1::text').get()
paragraphs = page.css('p::text').getall()
```
### 2. Stealthy Fetch (Anti-Bot/Cloudflare)
```python
from scrapling.fetchers import StealthyFetcher
StealthyFetcher.adaptive = True
page = StealthyFetcher.fetch('https://example.com', headless=True, solve_cloudflare=True)
```
### 3. Dynamic Fetch (Full Browser Automation)
```python
from scrapling.fetchers import DynamicFetcher
page = DynamicFetcher.fetch('https://example.com', headless=True, network_idle=True)
```
### 4. Adaptive Parsing (Survives Design Changes)
```python
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://example.com')
# First scrape - saves selectors
items = page.css('.product', auto_save=True)
# Later - if site changes, use adaptive=True to relocate
items = page.css('.product', adaptive=True)
```
### 5. Spider (Multiple Pages)
```python
from scrapling.spiders import Spider, Response
class MySpider(Spider):
name = "demo"
start_urls = ["https://example.com"]
concurrent_requests = 3
async def parse(self, response: Response):
for item in response.css('.item'):
yield {"item": item.css('h2::text').get()}
# Follow links
next_page = response.css('.next a')
if next_page:
yield response.follow(next_page[0].attrib['href'])
MySpider().start()
```
### 6. CLI Usage
```bash
# Simple fetch to file
scrapling extract get https://example.com content.html
# Stealthy fetch (bypass anti-bot)
scrapling extract stealthy-fetch https://example.com content.html
# Interactive shell
scrapling shell https://example.com
```
---
## Common Patterns
### Extract Article Content
```python
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://example.com/article')
# Try multiple selectors for title
title = (
page.css('[itemprop="headline"]::text').get() or
page.css('article h1::text').get() or
page.css('h1::text').get()
)
# Get paragraphs
content = page.css('article p::text, .article-body p::text').getall()
print(f"Title: {title}")
print(f"Paragraphs: {len(content)}")
```
### Research Multiple Pages
```python
from scrapling.spiders import Spider, Response
class ResearchSpider(Spider):
name = "research"
start_urls = ["https://news.ycombinator.com"]
concurrent_requests = 5
async def parse(self, response: Response):
for item in response.css('.titleline a::text').getall()[:10]:
yield {"title": item, "source": "HN"}
more = response.css('.morelink::attr(href)').get()
if more:
yield response.follow(more)
ResearchSpider().start()
```
### Crawl Entire Site (Easy Mode)
Auto-crawl all pages on a domain by following internal links:
```python
from scrapling.spiders import Spider, Response
from urllib.parse import urljoin, urlparse
class EasyCrawl(Spider):
"""Auto-crawl all pages on a domain."""
name = "easy_crawl"
start_urls = ["https://example.com"]
concurrent_requests = 3
def __init__(self):
super().__init__()
self.visited = set()
async def parse(self, response: Response):
# Extract content
yield {
'url': response.url,
'title': response.css('title::text').get(),
'h1': response.css('h1::text').get(),
}
# Follow internal links (limit to 50 pages)
if len(self.visited) >= 50:
return
self.visited.add(response.url)
links = response.css('a::attr(href)').getall()[:20]
for link in links:
full_url = urljoin(response.url, link)
if full_url not in self.visited:
yield response.follow(full_url)
# Usage
result = EasyCrawl()
result.start()
```
### Sitemap Crawl
Crawl pages from `sitemap.xml` (with fallback to link discovery):
```python
from scrapling.fetchers import Fetcher
from scrapling.spiders import Spider, Response
from urllib.parse import urljoin, urlparse
import re
def get_sitemap_urls(url: str, max_urls: int = 100) -> list:
"""Extract URLs from sitemap.xml - also checks robots.txt."""
parsed = urlparse(url)
base_url = f"{parsed.scheme}://{parsed.netloc}"
sitemap_urls = [
f"{base_url}/sitemap.xml",
f"{base_url}/sitemap-index.xml",
f"{base_url}/sitemap_index.xml",
f"{base_url}/sitemap-news.xml",
]
all_urls = []
# First check robots.txt for sitemap URL
try:
robots = Fetcher.get(f"{base_url}/robots.txt")
if robots.status == 200:
sitemap_in_robots = re.findall(r'Sitemap:\s*(\S+)', robots.text, re.IGNORECASE)
for sm in sitemap_in_robots:
sitemap_urls.insert(0, sm)
except:
pass
# Try each sitemap location
for sitemap_url in sitemap_urls:
try:
page = Fetcher.get(sitemap_url, timeout=10)
if page.status != 200:
continue
text = page.text
# Check if it's XML
if '<?xml' in text or '<urlset' in text or '<sitemapindex' in text:
urls = re.findall(r'<loc>([^<]+)</loc>', text)
all_urls.extend(urls[:max_urls])
print(f"Found {len(urls)} URLs in {sitemap_url}")
except:
continue
return list(set(all_urls))[:max_urls]
def crawl_from_sitemap(domain_url: str, max_pages: int = 50):
"""Crawl pages from sitemap."""
print(f"Fetching sitemap for {domain_url}...")
urls = get_sitemap_urls(domain_url)
if not urls:
print("No sitemap found. Use EasyCrawl instead!")
return []
print(f"Found {len(urls)} URLs, crawling first {max_pages}...")
results = []
for url in urls[:max_pages]:
try:
page = Fetcher.get(url, timeout=10)
results.append({
'url': url,
'status': page.status,
'title': page.css('title::text').get(),
})
except Exception as e:
results.append({'url': url, 'error': str(e)[:50]})
return results
# Usage
print("=== Sitemap Crawl ===")
results = crawl_from_sitemap('https://example.com', max_pages=10)
for r in results[:3]:
print(f" {r.get('title', r.get('error', 'N/A'))}")
# Alternative: Easy crawl all links
print("\n=== Easy Crawl (Link Discovery) ===")
result = EasyCrawl(start_urls=["https://example.com"], max_pages=10).start()
print(f"Crawled {len(result.items)} pages")
```
### Firecrawl-Style Crawl (Best of Both Worlds)
Inspired by Firecrawl's behavior - combines sitemap discovery with link following:
```python
from scrapling.fetchers import Fetcher
from scrapling.spiders import Spider, Response
from urllib.parse import urljoin, urlparse
import re
def firecrawl_crawl(url: str, max_pages: int = 50, use_sitemap: bool = True):
"""
Firecrawl-style crawling:
- use_sitemap=True: Discover URLs from sitemap first (default)
- use_sitemap=False: Only follow HTML links (like sitemap:"skip")
Matches Firecrawl's crawl behavior.
"""
parsed = urlparse(url)
domain = parsed.netloc
# ========== Method 1: Sitemap Discovery ==========
if use_sitemap:
print(f"[Firecrawl] Discovering URLs from sitemap...")
sitemap_urls = [
f"{url.rstrip('/')}/sitemap.xml",
f"{url.rstrip('/')}/sitemap-index.xml",
]
all_urls = []
# Try sitemaps
for sm_url in sitemap_urls:
try:
page = Fetcher.get(sm_url, timeout=15)
if page.status == 200:
# Handle bytes
text = page.body.decode('utf-8', errors='ignore') if isinstance(page.body, bytes) else str(page.body)
if '<urlset' in text:
urls = re.findall(r'<loc>([^<]+)</loc>', text)
all_urls.extend(urls[:max_pages])
print(f"[Firecrawl] Found {len(urls)} URLs in {sm_url}")
except:
continue
if all_urls:
print(f"[Firecrawl] Total: {len(all_urls)} URLs from sitemap")
# Crawl discovered URLs
results = []
for page_url in all_urls[:max_pages]:
try:
page = Fetcher.get(page_url, timeout=15)
results.append({
'url': page_url,
'status': page.status,
'title': page.css('title::text').get() if page.status == 200 else None,
})
except Exception as e:
results.append({'url': page_url, 'error': str(e)[:50]})
return results
# ========== Method 2: Link Discovery (sitemap: skip) ==========
print(f"[Firecrawl] Sitemap skip - using link discovery...")
class LinkCrawl(Spider):
name = "firecrawl_link"
start_urls = [url]
concurrent_requests = 3
def __init__(self):
super().__init__()
self.visited = set()
self.domain = domain
self.results = []
async def parse(self, response: Response):
if len(self.results) >= max_pages:
return
self.results.append({
'url': response.url,
'status': response.status,
'title': response.css('title::text').get(),
})
# Follow internal links
links = response.css('a::attr(href)').getall()[:20]
for link in links:
full_url = urljoin(response.url, link)
parsed_link = urlparse(full_url)
if parsed_link.netloc == self.domain and full_url not in self.visited:
self.visited.add(full_url)
if len(self.visited) < max_pages:
yield response.follow(full_url)
result = LinkCrawl()
result.start()
return result.results
# Usage
print("=== Firecrawl-Style (sitemap: include) ===")
results = firecrawl_crawl('https://www.cloudflare.com', max_pages=5, use_sitemap=True)
print(f"Crawled: {len(results)} pages")
print("\n=== Firecrawl-Style (sitemap: skip) ===")
results = firecrawl_crawl('https://example.com', max_pages=5, use_sitemap=False)
print(f"Crawled: {len(results)} pages")
```
### Handle Errors
```python
from scrapling.fetchers import Fetcher, StealthyFetcher
try:
page = Fetcher.get('https://example.com')
except Exception as e:
# Try stealth mode
page = StealthyFetcher.fetch('https://example.com', headless=True)
if page.status == 403:
print("Blocked - try StealthyFetcher")
elif page.status == 200:
print("Success!")
```
---
## Session Management
```python
from scrapling.fetchers import FetcherSession
with FetcherSession(impersonate='chrome') as session:
page = session.get('https://quotes.toscrape.com/', stealthy_headers=True)
quotes = page.css('.quote .text::text').getall()
```
### Multiple Session Types in Spider
```python
from scrapling.spiders import Spider, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession
class MultiSessionSpider(Spider):
name = "multi"
start_urls = ["https://example.com/"]
def configure_sessions(self, manager):
manager.add("fast", FetcherSession(impersonate="chrome"))
manager.add("stealth", AsyncStealthySession(headless=True), lazy=True)
async def parse(self, response: Response):
for link in response.css('a::attr(href)').getall():
if "protected" in link:
yield Request(link, sid="stealth")
else:
yield Request(link, sid="fast", callback=self.parse)
```
---
## Advanced Parsing & Navigation
```python
from scrapling.fetchers import Fetcher
page = Fetcher.get('https://quotes.toscrape.com/')
# Multiple selection methods
quotes = page.css('.quote') # CSS
quotes = page.xpath('//div[@class="quote"]') # XPath
quotes = page.find_all('div', class_='quote') # BeautifulSoup-style
# Navigation
first_quote = page.css('.quote')[0]
author = first_quote.css('.author::text').get()
parent = first_quote.parent
# Find similar elements
similar = first_quote.find_similar()
```
---
## Advanced: API Reverse Engineering
> "Web scraping is 80% reverse engineering."
This section covers advanced techniques to discover and replicate APIs directly from websites — often revealing data that's "hidden" behind paid APIs.
### 1. API Endpoint Discovery
Many websites load data via client-side requests. Use browser DevTools to find them:
**Steps:**
1. Open browser DevTools (F12)
2. Go to **Network** tab
3. Reload the page
4. Look for **XHR** or **Fetch** requests
5. Check if endpoints return JSON data
**What to look for:**
- Requests to `/api/*` endpoints
- Responses containing structured data (JSON)
- Same endpoints used on both free and paid sections
**Example pattern:**
```
# Found in Network tab:
GET https://api.example.com/v1/users/transactions
Response: {"data": [...], "pagination": {...}}
```
### 2. JavaScript Analysis
Auth tokens often generated client-side. Find them in `.js` files:
**Steps:**
1. In Network tab, look at **Initiator** column
2. Click the `.js` file making the request
3. Search for auth header name (e.g., `sol-aut`, `Authorization`, `X-API-Key`)
4. Find the function generating the token
**Common patterns:**
- Plain text function names: `generateToken()`, `createAuthHeader()`
- Obfuscated: Search for the header name directly
- Random string generation: `Math.random()`, `crypto.getRandomValues()`
### 3. Replicating Discovered APIs
Once you've found the endpoint and auth pattern:
```python
import requests
import random
import string
def generate_auth_token():
"""Replicate discovered token generation logic."""
chars = string.ascii_letters + string.digits
token = ''.join(random.choice(chars) for _ in range(40))
# Insert fixed string at random position
fixed = "B9dls0fK"
pos = random.randint(0, len(token))
return token[:pos] + fixed + token[pos:]
def scrape_api_endpoint(url):
"""Hit discovered API endpoint with replicated auth."""
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
'Accept': 'application/json',
'sol-aut': generate_auth_token(), # Replicate discovered header
}
response = requests.get(url, headers=headers)
return response.json()
```
### 4. Cloudscraper Bypass (Cloudflare)
For Cloudflare-protected endpoints, use `cloudscraper`:
```bash
pip install cloudscraper
```
```python
import cloudscraper
def create_scraper():
"""Create a cloudscraper session that bypasses Cloudflare."""
scraper = cloudscraper.create_scraper(
browser={
'browser': 'chrome',
'platform': 'windows',
'desktop': True
}
)
return scraper
# Usage
scraper = create_scraper()
response = scraper.get('https://api.example.com/endpoint')
data = response.json()
```
### 5. Complete API Replication Pattern
```python
import cloudscraper
import random
import string
import json
class APIReplicator:
"""Replicate discovered API from website."""
def __init__(self, base_url):
self.base_url = base_url
self.session = cloudscraper.create_scraper()
def generate_token(self, pattern="random"):
"""Replicate discovered token generation."""
if pattern == "solscan":
# 40-char random + fixed string at random position
chars = string.ascii_letters + string.digits
token = ''.join(random.choice(chars) for _ in range(40))
fixed = "B9dls0fK"
pos = random.randint(0, len(token))
return token[:pos] + fixed + token[pos:]
else:
# Generic random token
return ''.join(random.choices(string.ascii_letters + string.digits, k=32))
def get(self, endpoint, headers=None, auth_header=None, auth_pattern="random"):
"""Make API request with discovered auth."""
url = f"{self.base_url}{endpoint}"
# Build headers
request_headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
'Accept': 'application/json',
}
# Add discovered auth header
if auth_header:
request_headers[auth_header] = self.generate_token(auth_pattern)
# Merge custom headers
if headers:
request_headers.update(headers)
response = self.session.get(url, headers=request_headers)
return response
# Usage example
api = APIReplicator("https://api.solscan.io")
data = api.get(
"/account/transactions",
auth_header="sol-aut",
auth_pattern="solscan"
)
print(data)
```
### 6. Discovery Checklist
When approaching a new site:
| Step | Action | Tool |
|------|--------|------|
| 1 | Open DevTools Network tab | F12 |
| 2 | Reload page, filter by XHR/Fetch | Network filter |
| 3 | Look for JSON responses | Response tab |
| 4 | Check if same endpoint used for "premium" data | Compare requests |
| 5 | Find auth header in JS files | Initiator column |
| 6 | Extract token generation logic | JS debugger |
| 7 | Replicate in Python | Replicator class |
| 8 | Test against API | Run script |
---
## Brand Data Extraction (Firecrawl Alternative)
Extract brand data, colors, logos, and copy from any website:
```python
from scrapling.fetchers import Fetcher
from urllib.parse import urljoin
import re
def extract_brand_data(url: str) -> dict:
"""Extract structured brand data from any website - Firecrawl style."""
# Try stealth mode first (handles anti-bot)
try:
page = Fetcher.get(url)
except:
from scrapling.fetchers import StealthyFetcher
page = StealthyFetcher.fetch(url, headless=True)
# Helper to get text from element
def get_text(elements):
return elements[0].text if elements else None
# Helper to get attribute
def get_attr(elements, attr_name):
return elements[0].attrib.get(attr_name) if elements else None
# Brand name (try multiple selectors)
brand_name = (
get_text(page.css('[property="og:site_name"]')) or
get_text(page.css('h1')) or
get_text(page.css('title'))
)
# Tagline
tagline = (
get_text(page.css('[property="og:description"]')) or
get_text(page.css('.tagline')) or
get_text(page.css('.hero-text')) or
get_text(page.css('header h2'))
)
# Logo URL
logo_url = (
get_attr(page.css('[rel="icon"]'), 'href') or
get_attr(page.css('[rel="apple-touch-icon"]'), 'href') or
get_attr(page.css('.logo img'), 'src')
)
if logo_url and not logo_url.startswith('http'):
logo_url = urljoin(url, logo_url)
# Favicon
favicon = get_attr(page.css('[rel="icon"]'), 'href')
favicon_url = urljoin(url, favicon) if favicon else None
# OG Image
og_image = get_attr(page.css('[property="og:image"]'), 'content')
og_image_url = urljoin(url, og_image) if og_image else None
# Screenshot (using external service)
screenshot_url = f"https://image.thum.io/get/width/1200/crop/800/{url}"
# Description
description = (
get_text(page.css('[property="og:description"]')) or
get_attr(page.css('[name="description"]'), 'content')
)
# CTA text
cta_text = (
get_text(page.css('a[href*="signup"]')) or
get_text(page.css('.cta')) or
get_text(page.css('[class*="button"]'))
)
# Social links
social_links = {}
for platform in ['twitter', 'facebook', 'instagram', 'linkedin', 'youtube', 'github']:
link = get_attr(page.css(f'a[href*="{platform}"]'), 'href')
if link:
social_links[platform] = link
# Features (from feature grid/cards)
features = []
feature_cards = page.css('[class*="feature"], .feature-card, .benefit-item')
for card in feature_cards[:6]:
feature_text = get_text(card.css('h3, h4, p'))
if feature_text:
features.append(feature_text.strip())
return {
'brandName': brand_name,
'tagline': tagline,
'description': description,
'features': features,
'logoUrl': logo_url,
'faviconUrl': favicon_url,
'ctaText': cta_text,
'socialLinks': social_links,
'screenshotUrl': screenshot_url,
'ogImageUrl': og_image_url
}
# Usage
brand_data = extract_brand_data('https://example.com')
print(brand_data)
```
---
### Brand Data CLI
```bash
# Extract brand data using the Python function above
python3 -c "
import json
import sys
sys.path.insert(0, '/path/to/skill')
from brand_extraction import extract_brand_data
data = extract_brand_data('$URL')
print(json.dumps(data, indent=2))
"
```
---
## Feature Comparison
| Feature | Status | Notes |
|---------|--------|-------|
| Basic fetch | ✅ Working | Fetcher.get() |
| Stealthy fetch | ✅ Working | StealthyFetcher.fetch() |
| Dynamic fetch | ✅ Working | DynamicFetcher.fetch() |
| Adaptive parsing | ✅ Working | auto_save + adaptive |
| Spider crawling | ✅ Working | async def parse() |
| CSS selectors | ✅ Working | .css() |
| XPath | ✅ Working | .xpath() |
| Session management | ✅ Working | FetcherSession, StealthySession |
| Proxy rotation | ✅ Working | ProxyRotator class |
| CLI tools | ✅ Working | scrapling extract |
| Brand data extraction | ✅ Working | extract_brand_data() |
| API reverse engineering | ✅ Working | APIReplicator class |
| Cloudscraper bypass | ✅ Working | cloudscraper integration |
| Easy site crawl | ✅ Working | EasyCrawl class |
| Sitemap crawl | ✅ Working | get_sitemap_urls() |
| MCP server | ❌ Excluded | Not needed |
---
## Examples Tested
### IEEE Spectrum
```python
page = Fetcher.get('https://spectrum.ieee.org/...')
title = page.css('h1::text').get()
content = page.css('article p::text').getall()
```
✅ Works
### Hacker News
```python
page = Fetcher.get('https://news.ycombinator.com')
stories = page.css('.titleline a::text').getall()
```
✅ Works
### Example Domain
```python
page = Fetcher.get('https://example.com')
title = page.css('h1::text').get()
```
✅ Works
---
## 🔧 Quick Troubleshooting
| Issue | Solution |
|-------|----------|
| 403/429 Blocked | Use StealthyFetcher or cloudscraper |
| Cloudflare | Use StealthyFetcher or cloudscraper |
| JavaScript required | Use DynamicFetcher |
| Site changed | Use adaptive=True |
| Paid API exposed | Use API reverse engineering |
| Captcha | Cannot bypass - skip or use official API |
| Auth required | Do NOT bypass - use official API |
---
## Skill Graph
Related skills:
- [[content-research]] - Research workflow
- [[blogwatcher]] - RSS/feed monitoring
- [[youtube-watcher]] - Video content
- [[chirp]] - Twitter/X interactions
- [[newsletter-digest]] - Content summarization
- [[x-tweet-fetcher]] - X/Twitter (use instead of Scrapling)
---
## Changelog
### v1.0.8 (2026-02-25)
- **Added: Firecrawl-Style Crawl** - Combines sitemap discovery + link following
- **Added: use_sitemap parameter** - Matches Firecrawl's sitemap:"include"/"skip" behavior
- Verified: cloudflare.com returns 2,447 URLs from sitemap!
### v1.0.7 (2026-02-25)
- **Fixed: EasyCrawl Spider syntax** - Updated to work with scrapling's actual Spider API
- **Verified: Spider crawling works** - Tested and crawled 20+ pages from example.com
### v1.0.6 (2026-02-25)
- **Added: Easy Site Crawl** - Auto-crawl all pages on a domain with EasyCrawl spider
- **Added: Sitemap Crawl** - Extract URLs from sitemap.xml and crawl them
- Feature parity with Firecrawl for site crawling capabilities
### v1.0.5 (2026-02-25)
- **Enhanced: API Reverse Engineering methodology**
- Detailed step-by-step process from @paoloanzn's work
- Real Solscan case study with exact timeline
- Added: Step-by-step methodology section
- Added: Real example documentation (Solscan March 2025 vs Feb 2026)
- Added: Discovery checklist with 10 steps
- Documented: How to find auth headers in JS files
- Documented: Token generation pattern extraction
- Updated: Cloudscraper integration with multi-attempt pattern
- Verified: Solscan now patched (Cloudflare on both endpoints)
### v1.0.4 (2026-02-25)
- **Fixed: Brand Data Extraction API** - Corrected selectors for scrapling's Response object
- Fixed `.html` → `.text` / `.body`
- Fixed `.title()` → `page.css('title')`
- Fixed `.logo img::src` → `.logo img::attr(src)`
- Tested and verified working
### v1.0.3 (2026-02-25)
- **Added: API Reverse Engineering section**
- API Endpoint Discovery (Network tab analysis)
- JavaScript Analysis (finding auth logic)
- Cloudscraper integration for Cloudflare bypass
- Complete APIReplicator class
- Discovery checklist
- Added cloudscraper to installation
### v1.0.2 (2026-02-25)
- Synced with upstream GitHub README exactly
- Added Brand Data Extraction section
- Clean, core-only version
### v1.0.1 (2026-02-25)
- Synced with original Scrapling GitHub README
---
*Last updated: 2026-02-25*
中国金融机构数据爬取与分析综合 Skill v12.0.0。2334家/43大类机构名单、60+财经媒体源舆情、 A股报告/公告/研报、60+种查询多源金融数据(自动降级+缓存)、爬取调度与网页操作增强、 实时数据中枢、24 技术指标引擎、多源证据融合、110 MCP 工具 + 10 Resource + 14 Prompt。 🆕 v12.0.0 四条线(信息获取 / 打包下载 / 数据分析 / 登录): 编排:gather_info() 一次跑完"路由→抓取→去占位→质量分→能力边界→回执→打包", 8 类意图各带问法、输入输出、典型耗时与 not_for(说清哪件事做不了)。 诚实:反爬占位条目一律剔除并记账,空结果报 E_SOURCE_EMPTY,不把假数据判成 B 级成功。 交付:包内自带 manifest.json(逐文件 sha256),verify 查得出改动/缺失/多塞, 批量下载单条失败不中断整批,失败清单可直接只重下那几条。 登录:Cookie 由用户导出或扫码接管,档案化 + 健康度四档,对外输出全脱敏(值永不回显)。 何时触发:爬取机构、舆情爬取、东方财富、公告下载、实时行情、技术指标、统一查询、 交付包、批量重下、我已经登录了、登录态还有效吗(完整触发词见下方 auto_trigger)。
大宗商品价格数据查询技能。用于查询大宗商品期现货价格、宏观经济指标、产业链供需数据、进出口库存数据、金融市场数据等。当用户询问:(1) 期货/现货价格查询,如"螺纹钢价格"、"铜期货收盘价"; (2) 历史行情或趋势分析;(3) 区域或企业价格对比;(4) 宏观经济指标,如GDP、CPI;(5) 产业链上下游数据;(6) 多市场金融数据(期权、股票、债券、外汇);(7) 结算确认单生成时触发此技能。
将数据文件(CSV/TSV/TXT/XLSX/XLS/JSON)转化为独立交互式 ECharts HTML 图表——32 类图表、3 套主题,全离线无需 CDN。出图前可先取数据画像(--profile:每列 dtype/基数/缺失/样本/统计、实体粒度、候选洞察信号、疑似非数据行),让 agent 在用户没说想看什么时自己读懂数据并决定出什么图——哪列是 ID、哪列是维度、出什么图,全部由 agent 从画像的客观事实自行判断,技能不代笔;每张图内置事实卡(plot_stats + data_preview),其数值是解读时唯一可引用的数字。本技能负责出图与算事实,不产出书面报告或叙事文档。触发:用户提供或指向表格数据文件并要求出图;提到「画图/出图/可视化/生成图表/generate a chart/make a graph」;想把表格渲染成交互式 HTML 图表。不触发:要的是书面报告、叙事分析或幻灯片;没有数据文件;需要后端的仪表盘;图片/音频/视频等非表格内容。
AdMapix のデータ取得層で、広告やアプリの問い合わせを API 呼び出しに変え、生の JSON だけを返します。
论文配图一键生成:一条命令从 JSON/CSV/Excel 生成顶刊级学术图表,科研绘图 与期刊配图全场景覆盖。40 种图表(森林图/KM 生存曲线(自动风险表)/竞争风险累计发生率/ROC/校准 曲线/决策曲线 DCA/列线图/RCS 非线性剂量反应/亚组分析森林图/RMST 限制平均生存时间/Galbraith 径向图/火山图/雨云图/山脊图/哑铃图/游泳图(逐例时长+里程碑)/ 肿瘤缓解瀑布图/UpSet 交集/PRISMA 流程图/漏斗/BA 一致性/PCA/热图/箱线/小提琴/ 组合图/Cox 多因素回归森林图等),9 套配色(含色盲安全 Okabe-Ito 与期刊色板),9 种期刊投稿预设, YAML 流水线一条命令出全套论文图,一次渲染多格式导出(TIFF/PNG/PDF), PDF 重叠门禁,600dpi。纯本地运行,数据不出机,中文零配置。适用:写论文/ 投稿要配图、把数据变成可投稿图表、批量重出全套图。不适用:文本润色/翻译/ 改写(请用写作类工具);商业报表/仪表盘/海报装饰;无数据支撑的纯示意图。
Intelligent chart generation and data analysis skill. Reads user-supplied data files (CSV/Excel/JSON), analyzes data characteristics with LLM assistance, auto-recommends and generates interactive ECharts visualizations. Use when the user asks to analyze data, generate charts, create visualizations, or work with tabular data files.
全球智能投研工具(stock-researcher)v13.0:多源数据 + 概率预测 + 可导出报告。 核心能力: - 多源数据路由:按市场/类别选源,逐级降级;每个结果标注来源与时效等级;源不可用时明确标 unavailable - 全资产量化:股票/ETF/场外基金/指数/期货(期限结构、基差、持仓、折溢价)多因子分析 - 概率预测:1d/5d/1M/3M/6M 经验分布 + 分位数 + 方向概率 + 区间,kNN 邻域收缩融合全局分布; 样本不足自动弃权,可按历史命中样本(同桶)做概率校准 - 证据链简报:每条结论绑定来源、时间与数据质量,冲突信号保留而不抹平 - 报告导出:Word/PPT/PDF/Markdown/JSON/纯文本,纯标准库渲染 + 可选第三方引擎 - MCP Server 24 工具,Claude Code 可直接调用;知识库 12 适配器双向同步 三条铁律: - 数据不装:拿不到就报不可用,不拿 0 价、不拿空表冒充 - 预测不装:概率来自历史相似条件的经验分布,样本不足就说不足,不硬给方向 - 评分≠盈利保证:任何结论都带免责声明
Tavily 経由で公開ウェブを検索し、リンクと抜粋付きの構造化結果を返します。
データ月:2026-10 · ダウンロード数とスター数は公開スキルレジストリ(ClawHub、SkillHub)から月単位で集計しています。詳しくはデータの方法論をご覧ください。