jq 命令介绍

简介

jq 是一款命令行下处理 JSON 数据的工具。其可以接受标准输入，命令管道或者文件中的 JSON 数据，经过一系列的过滤器(filters)和表达式的转换后形成我们需要的数据结构并将结果输出到标准输出中。
jq 只能接受 well form 的 JSON 字符串作为输入内容。也就是说输入内容必须严格遵循 JSON 格式的标准。所有的属性名必须是以双引号包括的字符串。对象的最后一个属性的末尾或者数组的最后一个元素的末尾不能有逗号。否则 jq 会抛出无法解析 JSON 的错误。

安装

1. 使用软件包管理器安装

• redhat系Linux安装

yum install -y jq 

• Debian、Ubuntu、Linux Mint

sudo apt-get install jq

• SUSE Linux

sudo zypper install jq

2. 官网下载二进制文件

wget -O jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64 
chmod +x ./jq 
cp jq /usr/bin

示例

1. 原始数据

cat 1.txt

{"firstName":"John","lastName":"Smith","age":25,"address":{"streetAddress":"21 2nd Street","city":"New York","state":"NY","postalCode":"10021"},"phoneNumber":[{"type":"home","number":"212 555-1234"},{"type":"fax","number":"646 555-4567"}],"gender":{"type":"male"}}

2. 格式化输出

# cat 1.txt | jq .
{
  "firstName": "John",
  "lastName": "Smith",
  "age": 25,
  "address": {
    "streetAddress": "21 2nd Street",
    "city": "New York",
    "state": "NY",
    "postalCode": "10021"
  },
  "phoneNumber": [
    {
      "type": "home",
      "number": "212 555-1234"
    },
    {
      "type": "fax",
      "number": "646 555-4567"
    }
  ],
  "gender": {
    "type": "male"
  }

2、提取字段

获取地址 address 信息：

# cat 1.txt | jq .address
{
  "streetAddress": "21 2nd Street",
  "city": "New York",
  "state": "NY",
  "postalCode": "10021"
}

获取 address 信息中的邮编 postalCode：

# cat 1.txt | jq .address.postalCode
"10021"

3、解析数组中的元素

获取电话信息

# cat 1.txt | jq  .phoneNumber[]
{
  "type": "home",
  "number": "212 555-1234"
}
{
  "type": "fax",
  "number": "646 555-4567"
}

获取第一个电话号码

# cat 1.txt | jq  .phoneNumber[0]
{
  "type": "home",
  "number": "212 55

4、管道操作

假设我只想要家庭电话，而不是整个 JSON 数组数据，可以使用以下方式

# cat 1.txt | jq  '.phoneNumber[] | select(.type == "home") | .number'
"212 555-1234"

提取json指定字段转csv

5、将CSV数据转换为json

csv数据

cat  >> 1.csv  << EOF
1478,john,38
1529,lucy,25
1673,iris,22
EOF

转换格式

# jq -R -c 'split(",")|{"uid":.[0],"name":.[1],"age":.[2]|tonumber}' 1.csv
{"uid":"1478","name":"john","age":38}
{"uid":"1529","name":"lucy","age":25}
{"uid":"1673","name":"iris","age":22}

6、将json转csv

测试数据

cat >> jsonData.json << EOF
{"productId":"2723","click":60,"view":300,"deal":2,"day":"20130919"}
{"productId":"2728","click":130,"view":800,"deal":10,"day":"20130919"}
{"productId":"3609","click":50,"view":400,"deal":3,"day":"20130919"}
{"productId":"3783","click":375,"view":1200,"deal":50,"day":"20130919"}
{"productId":"3522","click":87,"view":573,"deal":15,"day":"20130919"}
EOF

转换格式

# jq -r '[.productId+"_"+.day,(.click|tostring),(.view|tostring),(.deal|tostring)]|join(",")' jsonData.json
2723_20130919,60,300,2
2728_20130919,130,800,10
3609_20130919,50,400,3
3783_20130919,375,1200,50
3522_20130919,87,573,15

参数说明

参考

官方指导
Parsing JSON with jq
IBM深度介绍
CSDN-Shell命令行处理JSON
JSON介绍
其他格式化json方法介绍