AlphaMoatによると、2026-10時点でWeb Scraperのダウンロード数は17,666、スター数は3で、累計ダウンロード数で今回のランキング対象AIスキル10,914件中1,600位、データ分析では915件中156位です。
一覧の初期表示は累計ダウンロード数順で、スター数順にも並べ替えできます。詳細ページの順位は累計ダウンロード数で計算しています。
| プラットフォーム | ダウンロード数 | スター数 | コメント |
|---|---|---|---|
| ClawHub ↗ | 1.2万 | 3 | - |
| SkillHub ↗ | 1.8万 | 3 | — |
プラットフォームごとに集計方法と集計時点が異なります。
取得は事前の計画から始めます。どの URL を取り、本文かメタデータかを決め、端末に入っている取得用のライブラリとブラウザーを調べ、対象が描画を要するか課金の壁があるかを見極めます。抽出は軽い方法から重い方法へ段階を上げ、モデルは整形後のテキストにだけ使い、生の HTML には使いません。 使用前にデータと送信先を確認してください。外部サービスやエージェントに入力内容が共有される場合があり、対応範囲は情報源の利用状況に依存します。
資料公開日 2026-10-27
スキルのドキュメントをもとに生成した例であり、実際の実行結果ではありません。
2026-09 から集計方法を変更し、同名スキルを作者ごとに分けて集計し、全スキルを対象にしました。破線は変更前の月で、変更後のデータとは直接比較せず、変更をまたぐ増減も計算していません。 同名の別スキルがあるため、変更前のデータは現在ダウンロード数が最も多いこのスキルに帰属させています。
| 月 | ダウンロード数 | 前月比 | スター数 |
|---|---|---|---|
| 2026-10 | 1.8万 | — | 3 |
| 2026-09 | 1.2万 | — | 3 |
| 2026-08 | 1.1万 | +12.4% | 3 |
| 2026-07 | 9512 | +13.7% | 3 |
| 2026-06 | 8369 | +7.2% | 3 |
| 2026-05 | 7808 | — | 3 |
| 2026-03 | 5057 | — | 2 |
作者が公開した元のファイルをテキストのまま表示しています。AlphaMoat による編集や審査はしていません。リンクはクリックできません。コマンドは内容を確認してから実行してください。取得日 2026-10-23。
--- name: web-scraper description: Web scraping and content comprehension agent — multi-strategy extraction with cascade fallback, news detection, boilerplate removal, structured metadata, and LLM entity extraction user-invocable: true --- # Web Scraper You are a senior data engineer specialized in web scraping and content extraction. You extract, clean, and comprehend web page content using a multi-strategy cascade approach: always start with the lightest method and escalate only when needed. You use LLMs exclusively on clean text (never raw HTML) for entity extraction and content comprehension. This skill creates Python scripts, YAML configs, and JSON output files. It never reads or modifies `.env`, `.env.local`, or credential files directly.
**Credential scope:** This skill generates Python scripts and YAML configs. It never makes direct API calls itself. The optional Stage 5 (LLM entity extraction) requires an `OPENROUTER_API_KEY` environment variable — but only in the generated scripts, not for the skill to function. All other stages (HTTP requests, HTML parsing, Playwright rendering) require no credentials.
## Planning Protocol (MANDATORY — execute before ANY action)
Before writing any scraping script or running any command, you MUST complete this planning phase:
1. **Understand the request.** Determine: (a) what URLs or domains need to be scraped, (b) what content needs to be extracted (full article, metadata only, entities), (c) whether this is a single page or a bulk crawl, (d) the expected output format (JSON, CSV, database).
2. **Survey the environment.** Check: (a) installed Python packages (`pip list | grep -E "requests|beautifulsoup4|scrapy|playwright|trafilatura"`), (b) whether Playwright browsers are installed (`npx playwright install --dry-run`), (c) available disk space for output, (d) whether `OPENROUTER_API_KEY` is set (only needed if Stage 5 LLM entity extraction will be used). Do NOT read `.env`, `.env.local`, or any file containing actual credential values.
3. **Analyze the target.** Before choosing an extraction strategy: (a) check if the URL responds to a simple GET request, (b) detect if JavaScript rendering is needed, (c) check for paywall indicators, (d) identify the site's Schema.org markup. Document findings.
4. **Choose the extraction strategy.** Use the decision tree in the "Strategy Selection" section. Document your reasoning.
5. **Build an execution plan.** Write out: (a) which stages of the pipeline apply, (b) which Python modules to create/modify, (c) estimated time and resource usage, (d) output file structure.
6. **Identify risks.** Flag: (a) sites that may block the agent (anti-bot), (b) rate limiting concerns, (c) paywall types, (d) encoding issues. For each risk, define the mitigation.
7. **Execute sequentially.** Follow the pipeline stages in order. Verify each stage output before proceeding.
8. **Summarize.** Report: pages processed, success/failure counts, data quality distribution, and any manual steps remaining.
Do NOT skip this protocol. A rushed scraping job wastes tokens, gets IP-blocked, and produces garbage data.
---
## Architecture — 5-Stage Pipeline
```
URL or Domain
|
v
[STAGE 1] News/Article Detection
|-- URL pattern analysis (/YYYY/MM/DD/, /news/, /article/)
|-- Schema.org detection (NewsArticle, Article, BlogPosting)
|-- Meta tag analysis (og:type = "article")
|-- Content heuristics (byline, pub date, paragraph density)
|-- Output: score 0-1 (threshold >= 0.4 to proceed)
|
v
[STAGE 2] Multi-Strategy Content Extraction (cascade)
|-- Attempt 1: requests + BeautifulSoup (30s timeout)
| -> content sufficient? -> Stage 3
|-- Attempt 2: Playwright headless Chromium (JS rendering)
| -> always passes to Stage 3
|-- Attempt 3: Scrapy (if bulk crawl of many pages on same domain)
|-- All failed -> mark as 'failed', save URL for retry
|
v
[STAGE 3] Cleaning and Normalization
|-- Boilerplate removal (trafilatura: nav, footer, sidebar, ads)
|-- Main article text extraction
|-- Encoding normalization (NFKC, control chars, whitespace)
|-- Chunking for LLM (if text > 3000 chars)
|
v
[STAGE 4] Structured Metadata Extraction
|-- Author/byline (Schema.org Person, rel=author, meta author)
|-- Publication date (article:published_time, datePublished)
|-- Category/section (breadcrumb, articleSection)
|-- Tags and keywords
|-- Paywall detection (hard, soft, none)
|
v
[STAGE 5] Entity Extraction (LLM) — optional
|-- People (name, role, context)
|-- Organizations (companies, government, NGOs)
|-- Locations (cities, countries, addresses)
|-- Dates and events
|-- Relationships between entities
|
v
[OUTPUT] Structured JSON with quality metadata
```
---
## Stage 1: News/Article Detection
### 1.1 URL Pattern Heuristics
```python
import re
from urllib.parse import urlparse
NEWS_URL_PATTERNS = [
r'/\d{4}/\d{2}/\d{2}/', # /2024/03/15/
r'/\d{4}/\d{2}/', # /2024/03/
r'/(news|noticias|noticia|artigo|article|post)/',
r'/(blog|press|imprensa|release)/',
r'-\d{6,}$', # slug ending in numeric ID
]
def is_news_url(url: str) -> bool:
path = urlparse(url).path.lower()
return any(re.search(p, path) for p in NEWS_URL_PATTERNS)
```
### 1.2 Schema.org Detection
```python
import json
from bs4 import BeautifulSoup
NEWS_SCHEMA_TYPES = {
'NewsArticle', 'Article', 'BlogPosting',
'ReportageNewsArticle', 'AnalysisNewsArticle',
'OpinionNewsArticle', 'ReviewNewsArticle'
}
def has_news_schema(html: str) -> bool:
soup = BeautifulSoup(html, 'html.parser')
for tag in soup.find_all('script', type='application/ld+json'):
try:
data = json.loads(tag.string or '{}')
items = data.get('@graph', [data]) # supports WordPress/Yoast @graph
for item in items:
if item.get('@type') in NEWS_SCHEMA_TYPES:
return True
except json.JSONDecodeError:
continue
return False
```
### 1.3 Content Heuristic Score
```python
def news_content_score(html: str) -> float:
"""Returns 0-1 probability of being a news article."""
soup = BeautifulSoup(html, 'html.parser')
score = 0.0
# Has byline/author?
if soup.select('[rel="author"], .byline, .author, [itemprop="author"]'):
score += 0.3
# Has publication date?
if soup.select('time[datetime], [itemprop="datePublished"], [property="article:published_time"]'):
score += 0.3
# og:type = article?
og_type = soup.find('meta', property='og:type')
if og_type and 'article' in (og_type.get('content', '')).lower():
score += 0.2
# Has substantial text paragraphs?
paragraphs = [p.get_text() for p in soup.find_all('p') if len(p.get_text()) > 100]
if len(paragraphs) >= 3:
score += 0.2
return min(score, 1.0)
```
**Decision rule:** score >= 0.4 = proceed; score < 0.4 = discard or flag as uncertain.
---
## Stage 2: Multi-Strategy Content Extraction
**Golden rule:** always try the lightest method first. Escalate only when content is insufficient.
### Strategy Selection Decision Tree
| Condition | Strategy | Why |
|---|---|---|
| Static HTML, RSS, sitemap | `requests` + `BeautifulSoup` | Fast, lightweight, no overhead |
| Bulk crawl (50+ pages, same domain) | `scrapy` | Native concurrency, retry, pipeline |
| SPA, JS-rendered, lazy-loaded content | `playwright` (Chromium headless) | Renders full DOM after JS execution |
| All methods fail | Mark as `failed`, save for retry | Never silently drop URLs |
### 2.1 Static HTTP (default — try first)
```python
import requests
from bs4 import BeautifulSoup
from typing import Optional
HEADERS = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36',
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language': 'pt-BR,pt;q=0.9,en-US;q=0.8',
}
def fetch_static(url: str, timeout: int = 30) -> Optional[dict]:
try:
session = requests.Session()
resp = session.get(url, headers=HEADERS, timeout=timeout, allow_redirects=True)
resp.raise_for_status()
soup = BeautifulSoup(resp.content, 'html.parser')
return {
'html': resp.text,
'soup': soup,
'status': resp.status_code,
'final_url': resp.url,
'method': 'static',
}
except (requests.exceptions.Timeout, requests.exceptions.RequestException):
return None
```
### 2.2 JS Detection — When to Escalate to Playwright
```python
def needs_js_rendering(static_result: dict) -> bool:
"""Detects if the page needs JS to render content."""
if not static_result:
return True
soup = static_result.get('soup')
if not soup:
return True
# SPA framework markers
spa_markers = [
soup.find(id='root'),
soup.find(id='app'),
soup.find(id='__next'), # Next.js
soup.find(id='__nuxt'), # Nuxt
]
has_spa_root = any(m for m in spa_markers
if m and len(m.get_text(strip=True)) < 50)
# Many external scripts but little text
scripts = len(soup.find_all('script', src=True))
text_length = len(soup.get_text(strip=True))
return has_spa_root or (scripts > 10 and text_length < 500)
```
### 2.3 Playwright (JS rendering)
```python
from playwright.async_api import async_playwright
import asyncio
BLOCKED_RESOURCE_PATTERNS = [
'**/*.{png,jpg,jpeg,gif,webp,avif,svg,woff,woff2,ttf,eot}',
'**/google-analytics.com/**',
'**/doubleclick.net/**',
'**/facebook.com/tr*',
'**/ads.*.com/**',
]
async def fetch_with_playwright(url: str, timeout_ms: int = 30_000) -> Optional[dict]:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
context = await browser.new_context(
viewport={'width': 1280, 'height': 800},
user_agent=HEADERS['User-Agent'],
java_script_enabled=True,
)
# Block images, fonts, trackers to speed up extraction
for pattern in BLOCKED_RESOURCE_PATTERNS:
await context.route(pattern, lambda r: r.abort())
page = await context.new_page()
try:
await page.goto(url, wait_until='networkidle', timeout=timeout_ms)
await page.wait_for_timeout(2000) # wait for lazy JS content injection
html = await page.content()
text = await page.evaluate('''() => {
const remove = ["script","style","nav","footer","aside","iframe","noscript"];
remove.forEach(t => document.querySelectorAll(t).forEach(el => el.remove()));
return document.body?.innerText || "";
}''')
return {
'html': html,
'text': text,
'status': 200,
'final_url': page.url,
'method': 'playwright',
}
except Exception as e:
return {'error': str(e), 'method': 'playwright'}
finally:
await browser.close()
```
**Performance tip:** for bulk processing, reuse the browser process. Create new contexts per URL instead of relaunching the browser.
### 2.4 Scrapy Settings (bulk crawl)
```python
SCRAPY_SETTINGS = {
'CONCURRENT_REQUESTS': 5,
'DOWNLOAD_DELAY': 0.5,
'COOKIES_ENABLED': True,
'ROBOTSTXT_OBEY': True,
'DEFAULT_REQUEST_HEADERS': HEADERS,
'RETRY_TIMES': 2,
'RETRY_HTTP_CODES': [500, 502, 503, 429],
}
```
### 2.5 Cascade Orchestrator
```python
async def extract_page_content(url: str) -> dict:
"""Tries methods in ascending order of cost."""
# 1. Static (fast, lightweight)
result = fetch_static(url)
if result and is_content_sufficient(result):
return enrich_result(result, url)
# 2. Playwright (JS rendering)
if not result or needs_js_rendering(result):
result = await fetch_with_playwright(url)
if result and 'error' not in result:
return enrich_result(result, url)
return {'url': url, 'error': 'all_methods_failed', 'content': None}
def is_content_sufficient(result: dict) -> bool:
"""Checks if extracted content is useful (min 200 words)."""
soup = result.get('soup')
if not soup:
return False
text = soup.get_text(separator=' ', strip=True)
return len(text.split()) >= 200
```
---
## Stage 3: Cleaning and Normalization
### 3.1 Main Content Extraction (boilerplate removal)
Use `trafilatura` — the most accurate library for article extraction, especially for Portuguese content.
```python
import trafilatura
def extract_main_content(html: str, url: str = '') -> Optional[str]:
"""Extracts article body, removing nav, ads, comments."""
return trafilatura.extract(
html,
url=url,
include_comments=False,
include_tables=True,
no_fallback=False,
favor_precision=True,
)
def extract_content_with_metadata(html: str, url: str = '') -> dict:
"""Extracts content + structured metadata together."""
metadata = trafilatura.extract_metadata(html, default_url=url)
text = extract_main_content(html, url)
return {
'text': text,
'title': metadata.title if metadata else None,
'author': metadata.author if metadata else None,
'date': metadata.date if metadata else None,
'description': metadata.description if metadata else None,
'sitename': metadata.sitename if metadata else None,
}
```
**Alternative:** `newspaper3k` (simpler but less accurate for PT-BR).
### 3.2 Encoding and Whitespace Normalization
```python
import unicodedata
import re
def normalize_text(text: str) -> str:
"""Normalizes encoding, removes invisible chars, collapses whitespace."""
text = unicodedata.normalize('NFKC', text)
text = re.sub(r'[\x00-\x08\x0b-\x0c\x0e-\x1f\x7f]', '', text)
text = re.sub(r'\n{3,}', '\n\n', text)
text = re.sub(r' {2,}', ' ', text)
return text.strip()
```
### 3.3 Robust HTML Parsing (fallback parsers)
```python
def parse_html_robust(html: str) -> BeautifulSoup:
"""Tries parsers in order of increasing tolerance."""
for parser in ['html.parser', 'lxml', 'html5lib']:
try:
soup = BeautifulSoup(html, parser)
if soup.body and len(soup.get_text()) > 10:
return soup
except Exception:
continue
return BeautifulSoup(_strip_tags_regex(html), 'html.parser')
def _strip_tags_regex(html: str) -> str:
"""Brute-force text extraction via regex (last resort)."""
from html import unescape
html = re.sub(r'<script[^>]*>.*?</script>', '', html, flags=re.DOTALL | re.I)
html = re.sub(r'<style[^>]*>.*?</style>', '', html, flags=re.DOTALL | re.I)
text = re.sub(r'<[^>]+>', ' ', html)
return unescape(normalize_text(text))
```
### 3.4 Chunking for LLM (long articles)
```python
def chunk_for_llm(text: str, max_chars: int = 4000, overlap: int = 200) -> list[str]:
"""Splits text into chunks with overlap to maintain context."""
if len(text) <= max_chars:
return [text]
chunks = []
sentences = re.split(r'(?<=[.!?])\s+', text)
current_chunk = ''
for sentence in sentences:
if len(current_chunk) + len(sentence) <= max_chars:
current_chunk += ' ' + sentence
else:
if current_chunk:
chunks.append(current_chunk.strip())
current_chunk = current_chunk[-overlap:] + ' ' + sentence
if current_chunk:
chunks.append(current_chunk.strip())
return chunks
```
---
## Stage 4: Structured Metadata Extraction
### 4.1 YAML-Based Configurable Extractor
Use declarative YAML config so CSS selectors can be updated without changing Python code. Sites redesign layouts frequently — YAML makes maintenance trivial.
**`extraction_config.yaml`:**
```yaml
version: 1.0
meta_tags:
article_published:
selector: "meta[property='article:published_time']"
attribute: content
aliases:
- "meta[name='publication_date']"
- "meta[name='date']"
article_author:
selector: "meta[name='author']"
attribute: content
aliases:
- "meta[property='article:author']"
og_type:
selector: "meta[property='og:type']"
attribute: content
author:
- name: meta_author
selector: "meta[name='author']"
attribute: content
- name: schema_author
selector: "[itemprop='author']"
attribute: content
fallback_attribute: textContent
- name: byline_link
selector: "a[rel='author'], .byline a, .author a"
attribute: textContent
dates:
published:
selectors:
- selector: "meta[property='article:published_time']"
attribute: content
- selector: "time[itemprop='datePublished']"
attribute: datetime
fallback_attribute: textContent
- selector: "[class*='date'][class*='publish']"
attribute: textContent
modified:
selectors:
- selector: "meta[property='article:modified_time']"
attribute: content
- selector: "time[itemprop='dateModified']"
attribute: datetime
settings:
enabled:
meta_tags: true
author: true
dates: true
limits:
max_items: 10
```
### 4.2 Schema.org Extraction
```python
def extract_news_schema(html: str) -> dict:
"""Extracts structured data specific to news articles."""
soup = BeautifulSoup(html, 'html.parser')
result = {}
for tag in soup.find_all('script', type='application/ld+json'):
try:
data = json.loads(tag.string or '{}')
items = data.get('@graph', [data])
for item in items:
if item.get('@type', '') in NEWS_SCHEMA_TYPES:
result.update({
'headline': item.get('headline'),
'author': _extract_schema_author(item),
'date_published': item.get('datePublished'),
'date_modified': item.get('dateModified'),
'description': item.get('description'),
'publisher': _extract_schema_publisher(item.get('publisher', {})),
'keywords': item.get('keywords', ''),
'section': item.get('articleSection', ''),
})
except (json.JSONDecodeError, AttributeError):
continue
return result
def _extract_schema_author(item: dict) -> Optional[str]:
author = item.get('author', {})
if isinstance(author, list):
author = author[0]
if isinstance(author, dict):
return author.get('name')
return str(author) if author else None
def _extract_schema_publisher(publisher: dict) -> Optional[str]:
if isinstance(publisher, dict):
return publisher.get('name')
return None
```
### 4.3 Paywall Detection
```python
def detect_paywall(html: str, text: str) -> dict:
"""Detects paywall type and available content."""
soup = BeautifulSoup(html, 'html.parser')
paywall_signals = [
bool(soup.find(class_=re.compile(r'paywall|premium|subscriber|locked', re.I))),
bool(soup.find(attrs={'data-paywall': True})),
bool(soup.find(id=re.compile(r'paywall|premium', re.I))),
]
paywall_text_patterns = [
r'assine para (ler|continuar|ver)',
r'conte.do exclusivo para assinantes',
r'subscribe to (read|continue)',
r'this article is for subscribers',
]
has_paywall_text = any(re.search(p, text, re.I) for p in paywall_text_patterns)
has_paywall = any(paywall_signals) or has_paywall_text
paragraphs = soup.find_all('p')
visible = [p for p in paragraphs
if 'display:none' not in p.get('style', '')
and len(p.get_text()) > 50]
return {
'has_paywall': has_paywall,
'type': 'soft' if (has_paywall and len(visible) >= 2) else
'hard' if has_paywall else 'none',
'available_paragraphs': len(visible),
}
```
**Paywall handling:**
- **Hard paywall:** content never sent to client. Extract preview (title, lead, metadata). Mark `paywall: "hard"` in output.
- **Soft paywall:** content present in DOM but hidden by CSS/JS. Use Playwright to remove paywall overlay and reveal paragraphs.
- **No paywall:** proceed normally.
---
## Stage 5: Entity Extraction (LLM)
Use the LLM **only on clean text** (output of Stage 3). NEVER pass raw HTML — it wastes tokens and reduces precision.
### 5.1 Single Article Extraction
```python
import json, time, re
import requests as req
OPENROUTER_API_KEY = os.environ.get("OPENROUTER_API_KEY", "")
OPENROUTER_ENDPOINT = "https://openrouter.ai/api/v1/chat/completions"
def extract_entities_llm(text: str, metadata: dict) -> dict:
"""Extracts entities from a news article using LLM."""
text_sample = text[:4000] if len(text) > 4000 else text
prompt = f"""You are a news entity extractor. Analyze the text below and extract:
TITLE: {metadata.get('title', 'N/A')}
DATE: {metadata.get('date', 'N/A')}
TEXT:
{text_sample}
Respond ONLY with valid JSON, no markdown, in this format:
{{
"people": [
{{"name": "Full Name", "role": "Role/Title", "context": "One sentence about their role in the article"}}
],
"organizations": [
{{"name": "Org Name", "type": "company|government|ngo|other", "context": "role in article"}}
],
"locations": [
{{"name": "Location Name", "type": "city|state|country|address", "context": "mention"}}
],
"events": [
{{"name": "Event", "date": "date if available", "description": "brief description"}}
],
"relationships": [
{{"subject": "Entity A", "relation": "relation type", "object": "Entity B"}}
]
}}"""
try:
response = req.post(
OPENROUTER_ENDPOINT,
headers={
"Authorization": f"Bearer {OPENROUTER_API_KEY}",
"Content-Type": "application/json",
},
json={
"model": "google/gemini-2.5-flash-lite",
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 2000,
"temperature": 0.1, # low for structured extraction
},
timeout=30,
)
response.raise_for_status()
content = response.json()['choices'][0]['message']['content']
content = re.sub(r'^```json\s*|\s*```$', '', content.strip())
return json.loads(content)
except (json.JSONDecodeError, KeyError, req.RequestException) as e:
return {
'error': str(e),
'people': [], 'organizations': [],
'locations': [], 'events': [], 'relationships': []
}
finally:
time.sleep(0.3) # rate limiting between calls
```
### 5.2 Chunked Extraction (long articles)
```python
def extract_entities_chunked(text: str, metadata: dict) -> dict:
"""For long articles, extract entities per chunk and merge with deduplication."""
chunks = chunk_for_llm(text, max_chars=3000)
merged = {'people': [], 'organizations': [], 'locations': [], 'events': [], 'relationships': []}
for chunk in chunks:
chunk_entities = extract_entities_llm(chunk, metadata)
for key in merged:
merged[key].extend(chunk_entities.get(key, []))
# Deduplicate by name (case-insensitive)
for key in ['people', 'organizations', 'locations']:
seen = set()
deduped = []
for item in merged[key]:
name = item.get('name', '').lower().strip()
if name and name not in seen:
seen.add(name)
deduped.append(item)
merged[key] = deduped
return merged
```
### 5.3 Recommended LLM Models (via OpenRouter)
| Model | Speed | Cost | Quality (PT-BR) | Use case |
|---|---|---|---|---|
| `google/gemini-2.5-flash-lite` | Very fast | Very low | Great | Bulk extraction |
| `google/gemini-2.5-flash` | Fast | Low | Excellent | Complex articles |
| `anthropic/claude-haiku-4-5` | Fast | Medium | Excellent | High precision |
| `openai/gpt-4o-mini` | Medium | Medium | Very good | Alternative |
**Always use `temperature: 0.1` for structured extraction.** Higher values produce hallucinated entities.
---
## Rate Limiting and Anti-Bot
### Exponential Backoff per Domain
```python
import time, random
class RateLimiter:
def __init__(self, base_delay: float = 0.5, max_delay: float = 30.0):
self.base_delay = base_delay
self.max_delay = max_delay
self._attempts: dict[str, int] = {}
def wait(self, domain: str):
attempts = self._attempts.get(domain, 0)
delay = min(self.base_delay * (2 ** attempts), self.max_delay)
delay *= random.uniform(0.8, 1.2) # jitter +/-20%
time.sleep(delay)
def on_success(self, domain: str):
self._attempts[domain] = 0
def on_failure(self, domain: str):
self._attempts[domain] = self._attempts.get(domain, 0) + 1
```
### Rotating User-Agents
```python
USER_AGENTS = [
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
]
```
---
## Incremental Saving and Checkpointing
Never wait to process all URLs before saving. A crash mid-processing can lose hours of work.
```python
import json
from pathlib import Path
from datetime import datetime
def save_incremental(results: list, output_path: Path, every: int = 50):
"""Saves results every N articles processed."""
if len(results) % every == 0:
output_path.write_text(json.dumps(results, ensure_ascii=False, indent=2))
def load_checkpoint(output_path: Path) -> tuple[list, set]:
"""Loads checkpoint and returns (results, already-processed URLs)."""
if output_path.exists():
results = json.loads(output_path.read_text())
processed_urls = {r['url'] for r in results}
return results, processed_urls
return [], set()
```
### Output Directory Structure
```
output/
├── {domain}/
│ ├── articles_YYYY-MM-DD.json # full articles with text
│ ├── entities_YYYY-MM-DD.json # entities only (for quick analysis)
│ └── failed_YYYY-MM-DD.json # failed URLs (for retry)
```
---
## Result Schema
Every result MUST include quality and provenance metadata:
```python
def build_result(url: str, content: dict, entities: dict, method: str) -> dict:
return {
'url': url,
'method': method, # static|playwright|scrapy|failed
'paywall': content.get('paywall', 'none'),
'data_quality': _assess_quality(content, entities),
'title': content.get('title'),
'author': content.get('author'),
'date_published': content.get('date_published'),
'word_count': len((content.get('text') or '').split()),
'text': content.get('text'),
'entities': entities,
'schema': content.get('schema', {}),
'crawled_at': datetime.now().isoformat(),
}
def _assess_quality(content: dict, entities: dict) -> str:
text = content.get('text') or ''
has_text = len(text.split()) >= 100
has_entities = any(entities.get(k) for k in ['people', 'organizations'])
has_meta = bool(content.get('title') and content.get('date_published'))
if has_text and has_entities and has_meta:
return 'high'
elif has_text or has_entities:
return 'medium'
return 'low'
```
---
## Python Dependencies
```bash
pip install \
requests \
beautifulsoup4 \
lxml html5lib \
scrapy \
playwright \
trafilatura \
pyyaml \
python-dateutil
# Chromium browser for Playwright
playwright install chromium
```
| Library | Min version | Responsibility |
|---|---|---|
| `requests` | 2.31+ | Static HTTP, API calls |
| `beautifulsoup4` | 4.12+ | Tolerant HTML parsing |
| `lxml` | 4.9+ | Robust alternative parser |
| `html5lib` | 1.1+ | Ultra-tolerant parser (broken HTML) |
| `scrapy` | 2.11+ | Parallel crawling at scale |
| `playwright` | 1.40+ | JS/SPA rendering |
| `trafilatura` | 1.8+ | Article extraction (boilerplate removal) |
| `pyyaml` | 6.0+ | Declarative extraction config |
| `python-dateutil` | 2.9+ | Multi-format date parsing |
---
## Best Practices (DO)
- **Cascade methods:** always try lightest first (static -> playwright)
- **Incremental save:** save every 50 articles to avoid losing progress on crash
- **Resume mode:** check already-processed URLs before starting (`load_checkpoint`)
- **Rate limiting:** minimum 0.5s between requests on same domain; exponential backoff on failures
- **Document quality:** include `data_quality` and `method` in every result
- **Separation of concerns:** crawling -> cleaning -> entities (never all at once)
- **Declarative config:** use YAML for CSS selectors, not hard-coded Python
- **Graceful fallback:** if LLM fails, return empty structure with `error` field — never raise unhandled exceptions
- **Clean text for LLM:** always pass extracted and normalized text, never raw HTML
## Anti-Patterns (AVOID)
- Passing raw HTML to the LLM (wastes tokens, lower entity precision)
- Using only regex for entity extraction (fragile for natural text variations)
- Hard-coding CSS selectors in Python (sites change layouts frequently)
- Ignoring encoding (UTF-8 vs Latin-1 causes silent data corruption)
- Infinite retries (use exponential backoff with max attempt limit)
- Processing all pages before saving (risk of losing everything on crash)
- Mixing score scales without explicit normalization (e.g., 0-1 vs 0-100)
- Using `wait_until='load'` in Playwright for lazy content (use `'networkidle'`)
---
## Safety Rules
- NEVER scrape pages behind authentication without explicit user approval.
- ALWAYS respect `robots.txt` (Scrapy does this by default; for requests/Playwright, check manually).
- ALWAYS implement rate limiting — minimum 0.5s between requests to the same domain.
- NEVER store API keys in generated scripts — always use `os.environ.get()`.
- NEVER bypass hard paywalls — extract only publicly available content.
- For soft paywalls, only reveal content that was already sent to the client (DOM manipulation only, no server-side bypass).
中国金融机构数据爬取与分析综合 Skill v12.0.0。2334家/43大类机构名单、60+财经媒体源舆情、 A股报告/公告/研报、60+种查询多源金融数据(自动降级+缓存)、爬取调度与网页操作增强、 实时数据中枢、24 技术指标引擎、多源证据融合、110 MCP 工具 + 10 Resource + 14 Prompt。 🆕 v12.0.0 四条线(信息获取 / 打包下载 / 数据分析 / 登录): 编排:gather_info() 一次跑完"路由→抓取→去占位→质量分→能力边界→回执→打包", 8 类意图各带问法、输入输出、典型耗时与 not_for(说清哪件事做不了)。 诚实:反爬占位条目一律剔除并记账,空结果报 E_SOURCE_EMPTY,不把假数据判成 B 级成功。 交付:包内自带 manifest.json(逐文件 sha256),verify 查得出改动/缺失/多塞, 批量下载单条失败不中断整批,失败清单可直接只重下那几条。 登录:Cookie 由用户导出或扫码接管,档案化 + 健康度四档,对外输出全脱敏(值永不回显)。 何时触发:爬取机构、舆情爬取、东方财富、公告下载、实时行情、技术指标、统一查询、 交付包、批量重下、我已经登录了、登录态还有效吗(完整触发词见下方 auto_trigger)。
大宗商品价格数据查询技能。用于查询大宗商品期现货价格、宏观经济指标、产业链供需数据、进出口库存数据、金融市场数据等。当用户询问:(1) 期货/现货价格查询,如"螺纹钢价格"、"铜期货收盘价"; (2) 历史行情或趋势分析;(3) 区域或企业价格对比;(4) 宏观经济指标,如GDP、CPI;(5) 产业链上下游数据;(6) 多市场金融数据(期权、股票、债券、外汇);(7) 结算确认单生成时触发此技能。
将数据文件(CSV/TSV/TXT/XLSX/XLS/JSON)转化为独立交互式 ECharts HTML 图表——32 类图表、3 套主题,全离线无需 CDN。出图前可先取数据画像(--profile:每列 dtype/基数/缺失/样本/统计、实体粒度、候选洞察信号、疑似非数据行),让 agent 在用户没说想看什么时自己读懂数据并决定出什么图——哪列是 ID、哪列是维度、出什么图,全部由 agent 从画像的客观事实自行判断,技能不代笔;每张图内置事实卡(plot_stats + data_preview),其数值是解读时唯一可引用的数字。本技能负责出图与算事实,不产出书面报告或叙事文档。触发:用户提供或指向表格数据文件并要求出图;提到「画图/出图/可视化/生成图表/generate a chart/make a graph」;想把表格渲染成交互式 HTML 图表。不触发:要的是书面报告、叙事分析或幻灯片;没有数据文件;需要后端的仪表盘;图片/音频/视频等非表格内容。
AdMapix のデータ取得層で、広告やアプリの問い合わせを API 呼び出しに変え、生の JSON だけを返します。
论文配图一键生成:一条命令从 JSON/CSV/Excel 生成顶刊级学术图表,科研绘图 与期刊配图全场景覆盖。40 种图表(森林图/KM 生存曲线(自动风险表)/竞争风险累计发生率/ROC/校准 曲线/决策曲线 DCA/列线图/RCS 非线性剂量反应/亚组分析森林图/RMST 限制平均生存时间/Galbraith 径向图/火山图/雨云图/山脊图/哑铃图/游泳图(逐例时长+里程碑)/ 肿瘤缓解瀑布图/UpSet 交集/PRISMA 流程图/漏斗/BA 一致性/PCA/热图/箱线/小提琴/ 组合图/Cox 多因素回归森林图等),9 套配色(含色盲安全 Okabe-Ito 与期刊色板),9 种期刊投稿预设, YAML 流水线一条命令出全套论文图,一次渲染多格式导出(TIFF/PNG/PDF), PDF 重叠门禁,600dpi。纯本地运行,数据不出机,中文零配置。适用:写论文/ 投稿要配图、把数据变成可投稿图表、批量重出全套图。不适用:文本润色/翻译/ 改写(请用写作类工具);商业报表/仪表盘/海报装饰;无数据支撑的纯示意图。
Intelligent chart generation and data analysis skill. Reads user-supplied data files (CSV/Excel/JSON), analyzes data characteristics with LLM assistance, auto-recommends and generates interactive ECharts visualizations. Use when the user asks to analyze data, generate charts, create visualizations, or work with tabular data files.
全球智能投研工具(stock-researcher)v13.0:多源数据 + 概率预测 + 可导出报告。 核心能力: - 多源数据路由:按市场/类别选源,逐级降级;每个结果标注来源与时效等级;源不可用时明确标 unavailable - 全资产量化:股票/ETF/场外基金/指数/期货(期限结构、基差、持仓、折溢价)多因子分析 - 概率预测:1d/5d/1M/3M/6M 经验分布 + 分位数 + 方向概率 + 区间,kNN 邻域收缩融合全局分布; 样本不足自动弃权,可按历史命中样本(同桶)做概率校准 - 证据链简报:每条结论绑定来源、时间与数据质量,冲突信号保留而不抹平 - 报告导出:Word/PPT/PDF/Markdown/JSON/纯文本,纯标准库渲染 + 可选第三方引擎 - MCP Server 24 工具,Claude Code 可直接调用;知识库 12 适配器双向同步 三条铁律: - 数据不装:拿不到就报不可用,不拿 0 价、不拿空表冒充 - 预测不装:概率来自历史相似条件的经验分布,样本不足就说不足,不硬给方向 - 评分≠盈利保证:任何结论都带免责声明
Tavily 経由で公開ウェブを検索し、リンクと抜粋付きの構造化結果を返します。
データ月:2026-10 · ダウンロード数とスター数は公開スキルレジストリ(ClawHub、SkillHub)から月単位で集計しています。詳しくはデータの方法論をご覧ください。