Complete Guide to jq in Linux: Manipulate JSON Data

The jq command on Linux is an indispensable tool for anyone working with JSON data. If you are a system administrator, developer, or data analyst, learning how to use jq can save you a lot of time and effort when dealing with JSON on the command line. In this complete guide, you will learn what jq is, how to install it, and how to use it to efficiently manipulate and transform JSON data.

What is jq on Linux?

jq is a powerful command-line tool used to process, query, and transform data in JSON format. It works similarly to tools like sed or awk, but is specifically designed to handle JSON, making it essential for tasks involving APIs, automation, and processing large volumes of data.

Why Use jq on Linux?

With the increased use of JSON as a data format, especially in APIs and system configurations, jq has become an essential tool. It allows you to manipulate JSON data directly in the terminal, without the need to write complex scripts in programming languages.

How to Install jq on Linux

Installing jq is simple and can be done directly through your Linux distribution's package manager. Here's how:

Installing on Ubuntu or Debian

sudo apt-get install jq

Installing on Fedora

sudo dnf install jq

Installing on CentOS/RHEL

sudo yum install jq

Basic jq Commands

Now that you have installed jq, let's explore some basic commands to get you started.

1. Filtering Simple Data with jq

Consider the following JSON:

{
  "nome": "João",
  "idade": 30,
  "cidade": "São Paulo"
}

To extract only the name:

echo '{"nome": "João", "idade": 30, "cidade": "São Paulo"}' | jq '.nome'

Output:

"João"

2. Formatting JSON with jq

jq can format JSON in a readable way:

echo '{"nome": "João", "idade": 30, "cidade": "São Paulo"}' | jq .

Output:

{
  "nome": "João",
  "idade": 30,
  "cidade": "São Paulo"
}

3. Filtering Arrays with jq

To list only the names from an array of JSON objects:

echo '[{"nome": "João", "idade": 30}, {"nome": "Maria", "idade": 25}, {"nome": "José", "idade": 40}]' | jq '.[].nome'

Output:

"João"
"Maria"
"José"

4. Applying Conditions with jq

Filter people with age greater than 30:

echo '[{"nome": "João", "idade": 30}, {"nome": "Maria", "idade": 25}, {"nome": "José", "idade": 40}]' | jq '.[] | select(.idade > 30)'

Output:

{
  "nome": "José",
  "idade": 40
}

5. Transforming JSON Structures

Create a comma-separated list of names:

echo '[{"nome": "João", "idade": 30}, {"nome": "Maria", "idade": 25}, {"nome": "José", "idade": 40}]' | jq -r '.[].nome' | paste -sd ","

Output:

João,Maria,José

6. Extracting Multiple Fields Simultaneously

If you need to extract more than one field from a JSON object, you can easily do it with jq:

echo '{"nome": "João", "idade": 30, "cidade": "São Paulo"}' | jq '{nome, cidade}'

Output:

{
  "nome": "João",
  "cidade": "São Paulo"
}

7. Renaming Fields in JSON

You can rename fields when processing JSON with jq. Here is how to rename nome to full_name:

echo '{"nome": "João", "idade": 30}' | jq '{full_name: .nome, idade: .idade}'

Output:

{
  "full_name": "João",
  "idade": 30
}

8. Working with Arrays and Transforming Data

Suppose you have an array of objects and want to extract the values of a specific field and transform them into a new structure:

echo '[{"produto": "A", "preco": 10}, {"produto": "B", "preco": 15}, {"produto": "C", "preco": 20}]' | jq '[.[] | {item: .produto, valor: .preco}]'

Output:

[
  {
    "item": "A",
    "valor": 10
  },
  {
    "item": "B",
    "valor": 15
  },
  {
    "item": "C",
    "valor": 20
  }
]

9. Filtering with Complex Conditions

You can create complex filters using logical operators to extract data that meets multiple conditions:

echo '[{"nome": "João", "idade": 30, "cidade": "São Paulo"}, {"nome": "Maria", "idade": 25, "cidade": "Rio de Janeiro"}, {"nome": "José", "idade": 40, "cidade": "São Paulo"}]' | jq '.[] | select(.idade > 30 and .cidade == "São Paulo")'

Output:

{
  "nome": "José",
  "idade": 40,
  "cidade": "São Paulo"
}

10. Calculating Statistics with jq

If you have an array of numbers, you can use jq to calculate the sum, average, and other aggregate values:

echo '[1, 2, 3, 4, 5]' | jq '{soma: add, media: (add / length), minimo: min, maximo: max}'

Output:

{
  "soma": 15,
  "media": 3,
  "minimo": 1,
  "maximo": 5
}

11. Extracting and Summing Values from a Specific Field

If you have an array of objects and need to sum the values of a specific field, such as prices:

echo '[{"produto": "A", "preco": 10}, {"produto": "B", "preco": 15}, {"produto": "C", "preco": 20}]' | jq '[.[] | .preco] | add'

Output:

45

12. Transforming JSON to CSV

You can convert JSON data to CSV format using jq, which is useful for exporting data to spreadsheets:

echo '[{"nome": "João", "idade": 30}, {"nome": "Maria", "idade": 25}, {"nome": "José", "idade": 40}]' | jq -r '.[] | [.nome, .idade] | @csv'

Output:

"João",30
"Maria",25
"José",40

13. Modifying and Updating Values

You can use jq to modify values in a JSON, for example, by increasing the age by 5 years:

echo '[{"nome": "João", "idade": 30}, {"nome": "Maria", "idade": 25}, {"nome": "José", "idade": 40}]' | jq 'map(.idade += 5)'

Output:

[
  {
    "nome": "João",
    "idade": 35
  },
  {
    "nome": "Maria",
    "idade": 30
  },
  {
    "nome": "José",
    "idade": 45
  }
]

Advanced Uses of jq in Linux

1. Extracting Data from APIs with jq and curl

Combine jq with curl to manipulate JSON data returned from APIs:

curl -s 'https://api.exemplo.com/dados' | jq '.resultados[] | {nome, valor}'

2. Summing Values with jq

To sum numeric values in an array:

echo '[1, 2, 3, 4, 5]' | jq 'add'

Output:

15

Conclusion: Why jq is Essential for IT Professionals

jq is not just a powerful tool; it is essential for any IT professional who needs to manipulate JSON regularly. Whether for processing API responses, automating tasks, or analyzing large volumes of data, jq offers a simple and efficient solution.

Frequently Asked Questions about jq on Linux

1. Does jq work on all Linux distributions?
Yes, jq is compatible with most Linux distributions.

2. Can I use jq to modify JSON files directly?
Yes, you can redirect jq output to modify JSON files.

3. Does jq support other languages besides JSON?
No, jq is specifically designed for JSON.

Additional Resources

• Official jq Documentation
• Practical Examples of jq Usage

---