gitgoodordietrying

CSV Data Pipeline

数据分析v1.0.0
作者:gitgoodordietrying更新于 2026-05-11csv-pipeline@clawhub_gitgoodordietrying/csv-pipeline
数据清洗报表生成数据洞察#CSV#Data Pipeline#Json

据 AlphaMoat 统计,截至 2026-10,CSV Data Pipeline 累计下载 10,729 次、获 4 个星标,按累计下载量在本次参与排名的 10,914 个 AI 技能中排名第 2,541 位,在「数据分析」分类(915 个)中排名第 240 位。

列表默认按累计下载量排序,也可按星标排序;详情页排名按累计下载量计算。

累计下载
1.1万
星标
4
全站下载量排名
#2,541
全站前 24% · 「数据分析」前 27%
平台累计下载星标评论
ClawHub ↗7,7702-
SkillHub ↗1.1万4—

各平台的计数口径不同,统计时间不一致。

这个技能是做什么的

做行过滤、数据集合并、分组聚合、格式互转、去重排序与汇总报告,也能逐行流式处理大文件。除 Python 3 之外不需要额外依赖,metadata 允许 python3、python 或 uv 任选其一,三个主流平台都在支持范围内。

资料发布于 2026-10-27

可以用来做什么

  • 把 CSV 转成 JSON 再按列汇总
  • 合并两份表格并去掉重复行

什么时候用

  • 手上是表格文件、需要清洗与统计时
  • 要做一次简单的数据抽取转换时

示例提示词

以下是根据技能文档生成的示例,不代表实际运行结果。

把这个 CSV 按第三列分组统计数量
合并这两个文件并删掉重复记录

月度数据趋势

累计下载1.1万
4,6427,9111.1万2026-03 · 累计下载 5,0932026-05 · 累计下载 6,2692026-06 · 累计下载 6,5362026-07 · 累计下载 7,0052026-08 · 累计下载 7,3732026-09 · 累计下载 7,8442026-10 · 累计下载 1.1万26-0326-0926-10
星标4
2342026-03 · 星标 22026-05 · 星标 22026-06 · 星标 42026-07 · 星标 42026-08 · 星标 42026-09 · 星标 42026-10 · 星标 426-0326-0926-10

2026-09 起统计方法调整:同名技能按作者分别统计,并覆盖全部技能。图中虚线为调整前的月份,与之后的数据不直接比较,也不计算跨越调整的增长。

月份累计下载环比星标
2026-101.1万—4
2026-097,844—4
2026-087,373+5.3%4
2026-077,005+7.2%4
2026-066,536+4.3%4
2026-056,269—2
2026-035,093—2

SKILL.md 原文

以下是作者发布的原始文件,按纯文本显示,本站未做修改或审核;其中的链接不可点击,命令请核实后再运行。抓取于 2026-10-23。

12 KB · ClawHub v1.0.0
---
name: csv-pipeline
description: Process, transform, analyze, and report on CSV and JSON data files. Use when the user needs to filter rows, join datasets, compute aggregates, convert formats, deduplicate, or generate summary reports from tabular data. Works with any CSV, TSV, or JSON Lines file.
metadata: {"clawdbot":{"emoji":"📊","requires":{"anyBins":["python3","python","uv"]},"os":["linux","darwin","win32"]}}
---

# CSV Data Pipeline

Process tabular data (CSV, TSV, JSON, JSON Lines) using standard command-line tools and Python. No external dependencies required beyond Python 3.
展开全部(共 434 行)
## When to Use

- User provides a CSV/TSV/JSON file and asks to analyze, transform, or report on it
- Joining, filtering, grouping, or aggregating tabular data
- Converting between formats (CSV to JSON, JSON to CSV, etc.)
- Deduplicating, sorting, or cleaning messy data
- Generating summary statistics or reports
- ETL workflows: extract from one format, transform, load into another

## Quick Operations with Standard Tools

### Inspect

```bash
# Preview first rows
head -5 data.csv

# Count rows (excluding header)
tail -n +2 data.csv | wc -l

# Show column headers
head -1 data.csv

# Count unique values in a column (column 3)
tail -n +2 data.csv | cut -d',' -f3 | sort -u | wc -l
```

### Filter with `awk`

```bash
# Filter rows where column 3 > 100
awk -F',' 'NR==1 || $3 > 100' data.csv > filtered.csv

# Filter rows matching a pattern in column 2
awk -F',' 'NR==1 || $2 ~ /pattern/' data.csv > matched.csv

# Sum column 4
awk -F',' 'NR>1 {sum += $4} END {print sum}' data.csv
```

### Sort and Deduplicate

```bash
# Sort by column 2 (numeric)
head -1 data.csv > sorted.csv && tail -n +2 data.csv | sort -t',' -k2 -n >> sorted.csv

# Deduplicate by all columns
head -1 data.csv > deduped.csv && tail -n +2 data.csv | sort -u >> deduped.csv

# Deduplicate by specific column (keep first occurrence)
awk -F',' '!seen[$2]++' data.csv > deduped.csv
```

## Python Operations (for complex transforms)

### Read and Inspect

```python
import csv, json, sys
from collections import Counter

def read_csv(path, delimiter=','):
    """Read CSV/TSV into list of dicts."""
    with open(path, newline='', encoding='utf-8') as f:
        return list(csv.DictReader(f, delimiter=delimiter))

def write_csv(rows, path, delimiter=','):
    """Write list of dicts to CSV."""
    if not rows:
        return
    with open(path, 'w', newline='', encoding='utf-8') as f:
        writer = csv.DictWriter(f, fieldnames=rows[0].keys(), delimiter=delimiter)
        writer.writeheader()
        writer.writerows(rows)

# Quick stats
data = read_csv('data.csv')
print(f"Rows: {len(data)}")
print(f"Columns: {list(data[0].keys())}")
for col in data[0]:
    non_empty = sum(1 for r in data if r[col].strip())
    print(f"  {col}: {non_empty}/{len(data)} non-empty")
```

### Filter and Transform

```python
# Filter rows
filtered = [r for r in data if float(r['amount']) > 100]

# Add computed column
for r in data:
    r['total'] = str(float(r['price']) * int(r['quantity']))

# Rename columns
renamed = [{('new_name' if k == 'old_name' else k): v for k, v in r.items()} for r in data]

# Type conversion
for r in data:
    r['amount'] = float(r['amount'])
    r['date'] = r['date'].strip()
```

### Group and Aggregate

```python
from collections import defaultdict

def group_by(rows, key):
    """Group rows by a column value."""
    groups = defaultdict(list)
    for r in rows:
        groups[r[key]].append(r)
    return dict(groups)

def aggregate(rows, group_col, agg_col, func='sum'):
    """Aggregate a column by groups."""
    groups = group_by(rows, group_col)
    results = []
    for name, group in sorted(groups.items()):
        values = [float(r[agg_col]) for r in group if r[agg_col].strip()]
        if func == 'sum':
            agg = sum(values)
        elif func == 'avg':
            agg = sum(values) / len(values) if values else 0
        elif func == 'count':
            agg = len(values)
        elif func == 'min':
            agg = min(values) if values else 0
        elif func == 'max':
            agg = max(values) if values else 0
        results.append({group_col: name, f'{func}_{agg_col}': str(agg), 'count': str(len(group))})
    return results

# Example: sum revenue by category
summary = aggregate(data, 'category', 'revenue', 'sum')
write_csv(summary, 'summary.csv')
```

### Join Datasets

```python
def inner_join(left, right, on):
    """Inner join two datasets on a key column."""
    right_index = {}
    for r in right:
        key = r[on]
        if key not in right_index:
            right_index[key] = []
        right_index[key].append(r)

    results = []
    for lr in left:
        key = lr[on]
        if key in right_index:
            for rr in right_index[key]:
                merged = {**lr}
                for k, v in rr.items():
                    if k != on:
                        merged[k] = v
                results.append(merged)
    return results

def left_join(left, right, on):
    """Left join: keep all left rows, fill missing right with empty."""
    right_index = {}
    right_cols = set()
    for r in right:
        key = r[on]
        right_cols.update(r.keys())
        if key not in right_index:
            right_index[key] = []
        right_index[key].append(r)
    right_cols.discard(on)

    results = []
    for lr in left:
        key = lr[on]
        if key in right_index:
            for rr in right_index[key]:
                merged = {**lr}
                for k, v in rr.items():
                    if k != on:
                        merged[k] = v
                results.append(merged)
        else:
            merged = {**lr}
            for col in right_cols:
                merged[col] = ''
            results.append(merged)
    return results

# Example
orders = read_csv('orders.csv')
customers = read_csv('customers.csv')
joined = left_join(orders, customers, on='customer_id')
write_csv(joined, 'orders_with_customers.csv')
```

### Deduplicate

```python
def deduplicate(rows, key_cols=None):
    """Remove duplicate rows. If key_cols specified, dedupe by those columns only."""
    seen = set()
    unique = []
    for r in rows:
        if key_cols:
            key = tuple(r[c] for c in key_cols)
        else:
            key = tuple(sorted(r.items()))
        if key not in seen:
            seen.add(key)
            unique.append(r)
    return unique

# Deduplicate by email column
clean = deduplicate(data, key_cols=['email'])
```

## Format Conversion

### CSV to JSON

```python
import json, csv

with open('data.csv', newline='', encoding='utf-8') as f:
    rows = list(csv.DictReader(f))

# Array of objects
with open('data.json', 'w') as f:
    json.dump(rows, f, indent=2)

# JSON Lines (one object per line, streamable)
with open('data.jsonl', 'w') as f:
    for row in rows:
        f.write(json.dumps(row) + '\n')
```

### JSON to CSV

```python
import json, csv

with open('data.json') as f:
    rows = json.load(f)

with open('data.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.DictWriter(f, fieldnames=rows[0].keys())
    writer.writeheader()
    writer.writerows(rows)
```

### JSON Lines to CSV

```python
import json, csv

rows = []
with open('data.jsonl') as f:
    for line in f:
        if line.strip():
            rows.append(json.loads(line))

with open('data.csv', 'w', newline='', encoding='utf-8') as f:
    all_keys = set()
    for r in rows:
        all_keys.update(r.keys())
    writer = csv.DictWriter(f, fieldnames=sorted(all_keys))
    writer.writeheader()
    writer.writerows(rows)
```

### TSV to CSV

```bash
tr '\t' ',' < data.tsv > data.csv
```

## Data Cleaning Patterns

### Fix common CSV issues

```python
def clean_csv(rows):
    """Clean common CSV data quality issues."""
    cleaned = []
    for r in rows:
        clean_row = {}
        for k, v in r.items():
            # Strip whitespace from keys and values
            k = k.strip()
            v = v.strip() if isinstance(v, str) else v
            # Normalize empty values
            if v in ('', 'N/A', 'n/a', 'NA', 'null', 'NULL', 'None', '-'):
                v = ''
            # Normalize boolean values
            if v.lower() in ('true', 'yes', '1', 'y'):
                v = 'true'
            elif v.lower() in ('false', 'no', '0', 'n'):
                v = 'false'
            clean_row[k] = v
        cleaned.append(clean_row)
    return cleaned
```

### Validate data types

```python
def validate_rows(rows, schema):
    """
    Validate rows against a schema.
    schema: dict of column_name -> 'int'|'float'|'date'|'email'|'str'
    Returns (valid_rows, error_rows)
    """
    import re
    valid, errors = [], []
    for i, r in enumerate(rows):
        errs = []
        for col, dtype in schema.items():
            val = r.get(col, '').strip()
            if not val:
                continue
            if dtype == 'int':
                try:
                    int(val)
                except ValueError:
                    errs.append(f"{col}: '{val}' not int")
            elif dtype == 'float':
                try:
                    float(val)
                except ValueError:
                    errs.append(f"{col}: '{val}' not float")
            elif dtype == 'email':
                if not re.match(r'^[^@]+@[^@]+\.[^@]+$', val):
                    errs.append(f"{col}: '{val}' not email")
            elif dtype == 'date':
                if not re.match(r'^\d{4}-\d{2}-\d{2}', val):
                    errs.append(f"{col}: '{val}' not YYYY-MM-DD")
        if errs:
            errors.append({'row': i + 2, 'errors': errs, 'data': r})
        else:
            valid.append(r)
    return valid, errors

# Usage
valid, bad = validate_rows(data, {'amount': 'float', 'email': 'email', 'date': 'date'})
print(f"Valid: {len(valid)}, Errors: {len(bad)}")
for e in bad[:5]:
    print(f"  Row {e['row']}: {e['errors']}")
```

## Generating Reports

### Summary report as Markdown

```python
def generate_report(data, title, group_col, value_col):
    """Generate a Markdown summary report."""
    lines = [f"# {title}", f"", f"**Total rows**: {len(data)}", ""]

    # Group summary
    groups = group_by(data, group_col)
    lines.append(f"## By {group_col}")
    lines.append("")
    lines.append(f"| {group_col} | Count | Sum | Avg | Min | Max |")
    lines.append("|---|---|---|---|---|---|")

    for name in sorted(groups):
        vals = [float(r[value_col]) for r in groups[name] if r[value_col].strip()]
        if vals:
            lines.append(f"| {name} | {len(vals)} | {sum(vals):.2f} | {sum(vals)/len(vals):.2f} | {min(vals):.2f} | {max(vals):.2f} |")

    lines.append("")
    lines.append(f"*Generated from {len(data)} rows*")
    return '\n'.join(lines)

report = generate_report(data, "Sales Summary", "category", "revenue")
with open('report.md', 'w') as f:
    f.write(report)
```

## Large File Handling

For files too large to load into memory at once:

```python
def stream_process(input_path, output_path, transform_fn, delimiter=','):
    """Process a CSV row-by-row without loading entire file."""
    with open(input_path, newline='', encoding='utf-8') as fin, \
         open(output_path, 'w', newline='', encoding='utf-8') as fout:
        reader = csv.DictReader(fin, delimiter=delimiter)
        writer = None
        for row in reader:
            result = transform_fn(row)
            if result is None:
                continue  # Skip row
            if writer is None:
                writer = csv.DictWriter(fout, fieldnames=result.keys(), delimiter=delimiter)
                writer.writeheader()
            writer.writerow(result)

# Example: filter and transform in streaming fashion
def process_row(row):
    if float(row.get('amount', 0) or 0) < 10:
        return None  # Skip small amounts
    row['amount_usd'] = str(float(row['amount']) * 1.0)  # Add computed field
    return row

stream_process('big_file.csv', 'output.csv', process_row)
```

## Tips

- Always check encoding: `file -i data.csv` or open with `encoding='utf-8-sig'` for BOM files
- For Excel exports with commas in values, the CSV module handles quoting automatically
- Use `json.dumps(ensure_ascii=False)` for international characters
- Pipe-delimited files: use `delimiter='|'` in csv.reader/writer
- For very large aggregations, consider `sqlite3` which Python includes:
  ```bash
  sqlite3 :memory: ".mode csv" ".import data.csv t" "SELECT category, SUM(amount) FROM t GROUP BY category;"
  ```

「数据分析」分类下的热门技能

全能金融爬虫数据分析

中国金融机构数据爬取与分析综合 Skill v12.0.0。2334家/43大类机构名单、60+财经媒体源舆情、 A股报告/公告/研报、60+种查询多源金融数据(自动降级+缓存)、爬取调度与网页操作增强、 实时数据中枢、24 技术指标引擎、多源证据融合、110 MCP 工具 + 10 Resource + 14 Prompt。 🆕 v12.0.0 四条线(信息获取 / 打包下载 / 数据分析 / 登录): 编排:gather_info() 一次跑完"路由→抓取→去占位→质量分→能力边界→回执→打包", 8 类意图各带问法、输入输出、典型耗时与 not_for(说清哪件事做不了)。 诚实:反爬占位条目一律剔除并记账,空结果报 E_SOURCE_EMPTY,不把假数据判成 B 级成功。 交付:包内自带 manifest.json(逐文件 sha256),verify 查得出改动/缺失/多塞, 批量下载单条失败不中断整批,失败清单可直接只重下那几条。 登录:Cookie 由用户导出或扫码接管,档案化 + 健康度四档,对外输出全脱敏(值永不回显)。 何时触发:爬取机构、舆情爬取、东方财富、公告下载、实时行情、技术指标、统一查询、 交付包、批量重下、我已经登录了、登录态还有效吗(完整触发词见下方 auto_trigger)。

↓ 77.0万★ 288user_9d5a2a39
钢联AI - 智能问数数据分析

大宗商品价格数据查询技能。用于查询大宗商品期现货价格、宏观经济指标、产业链供需数据、进出口库存数据、金融市场数据等。当用户询问:(1) 期货/现货价格查询,如"螺纹钢价格"、"铜期货收盘价"; (2) 历史行情或趋势分析;(3) 区域或企业价格对比;(4) 宏观经济指标,如GDP、CPI;(5) 产业链上下游数据;(6) 多市场金融数据(期权、股票、债券、外汇);(7) 结算确认单生成时触发此技能。

↓ 70.5万★ 6u_9ed3401d
smart-charts数据分析

32种图表,3 个主题,2步生成: 1.上传数据 — 将 CSV / Excel / JSON 等格式的数据文件拖入对话框; 2.查看结果 — 交互式图表(HTML),可保存图表为图片.

↓ 60.5万★ 135user_5b28ea14
AdMapix数据分析

AdMapix 的取数层:把广告与应用相关的查询转成接口调用,只回原始 JSON。

↓ 57.5万★ 304fly0pants
论文配图一键生成数据分析

论文配图一键生成:一条命令从 JSON/CSV/Excel 生成顶刊级学术图表,科研绘图 与期刊配图全场景覆盖。40 种图表(森林图/KM 生存曲线(自动风险表)/竞争风险累计发生率/ROC/校准 曲线/决策曲线 DCA/列线图/RCS 非线性剂量反应/亚组分析森林图/RMST 限制平均生存时间/Galbraith 径向图/火山图/雨云图/山脊图/哑铃图/游泳图(逐例时长+里程碑)/ 肿瘤缓解瀑布图/UpSet 交集/PRISMA 流程图/漏斗/BA 一致性/PCA/热图/箱线/小提琴/ 组合图/Cox 多因素回归森林图等),9 套配色(含色盲安全 Okabe-Ito 与期刊色板),9 种期刊投稿预设, YAML 流水线一条命令出全套论文图,一次渲染多格式导出(TIFF/PNG/PDF), PDF 重叠门禁,600dpi。纯本地运行,数据不出机,中文零配置。适用:写论文/ 投稿要配图、把数据变成可投稿图表、批量重出全套图。不适用:文本润色/翻译/ 改写(请用写作类工具);商业报表/仪表盘/海报装饰;无数据支撑的纯示意图。

↓ 42.1万★ 83user_481397b5
smart-charts数据分析

Intelligent chart generation and data analysis skill. Reads user-supplied data files (CSV/Excel/JSON), analyzes data characteristics with LLM assistance, auto-recommends and generates interactive ECharts visualizations. Use when the user asks to analyze data, generate charts, create visualizations, or work with tabular data files.

↓ 25.7万★ 1user_814dbe54
股票/基金/期货量化分析工具数据分析

全球智能投研工具(stock-researcher)v13.0:多源数据 + 概率预测 + 可导出报告。 核心能力: - 多源数据路由:按市场/类别选源,逐级降级;每个结果标注来源与时效等级;源不可用时明确标 unavailable - 全资产量化:股票/ETF/场外基金/指数/期货(期限结构、基差、持仓、折溢价)多因子分析 - 概率预测:1d/5d/1M/3M/6M 经验分布 + 分位数 + 方向概率 + 区间,kNN 邻域收缩融合全局分布; 样本不足自动弃权,可按历史命中样本(同桶)做概率校准 - 证据链简报:每条结论绑定来源、时间与数据质量,冲突信号保留而不抹平 - 报告导出:Word/PPT/PDF/Markdown/JSON/纯文本,纯标准库渲染 + 可选第三方引擎 - MCP Server 24 工具,Claude Code 可直接调用;知识库 12 适配器双向同步 三条铁律: - 数据不装:拿不到就报不可用,不拿 0 价、不拿空表冒充 - 预测不装:概率来自历史相似条件的经验分布,样本不足就说不足,不硬给方向 - 评分≠盈利保证:任何结论都带免责声明

↓ 19.2万★ 147user_9d5a2a39
Tavily Search数据分析

走 Tavily 检索公开网页,返回带链接与摘要的结构化结果。

↓ 16.5万★ 286Xiaoran Li

gitgoodordietrying 的其他技能

数据月份:2026-10 · 下载/星标数聚合自公开技能注册库(ClawHub、SkillHub),按月更新。口径说明见数据方法。