What is jq?
jq is a powerful expression language for filtering and transforming structured JSON data. It's available for the command line and a variety of programming languages, including Java, Python, and Javascript.
Helpful Resources
- jqplay: A playground for jq 1.6
- jq Manual: has runnable examples of each operator
- jq Homepage
- jq tag on StackOverflow
Brick Arguments
The jq brick takes two input arguments:
data
: an object, array, or single value which is passed to jqfilter
: the jq "filter", which controls how to filter/transform the data
Brick Argument Examples
Scenario #1: passing a variable for data
You can pass a variable or template directly to data:
id: "@pixiebrix/jq"
config:
# @apiResult = {items: []}
data: "@apiResult"
filter: ".items"
Scenario #2: passing multiple variables to data
You can also pass multiple variables by passing a dictionary to data, including using template syntax:
id: "@pixiebrix/jq"
config:
data:
# @apiResult = {items: [{text: "foo"}]}
result: "@apiResult"
# @input = {query: "foo"}
query: "{{ @input.query }}!"
filter: ".query as $query | [.result.items[] | select(.text == $query)]"
Common Pitfalls
- jq only has access to the data you passed in via the
data
argument. You should not use@variable
syntax in the jqfilter
- jq uses a
.
to refer to its input (the data you passed to the brick) - jq can return multiple results. However, PixieBrix expects a single result from jq. Therefore, you must wrap an expression that returns multiple results in Array construction
[]
operator
Filter Examples
Creating a new object from the input
{foo: 42, bar: .}
Parsing and summing values
.transactions | map(.amount | capture("(?<val>\\d+) USD") | .val | tonumber) | add
Filtering an array
Assign the query to a variable so you can use it in the select
clause.
NOTE: jq's select
operator returns multiple results. Wrap the expression in an Array Construction []
operator to return the results as a single array
.query as $query | [.items[] | select(.text == $query)]
Looking up a value in a pre-defined map (Workshop Example)
Pass the pre-defined map as an input to the jq brick. Then, assign the map to a variable (e.g., $m
) so you can lookup up the value later in the filter:
data:
# ["A", "B", "C"]
items: "@items"
mapping:
A: Alpha
B: Bravo
C: Charlie
filter: ".mapping as $m | .items | map($m[.])"
You can use jq's //
defaulting operator to handle elements that aren't in the map. E.g.,
filter: '.mapping as $m | .items | map($m[.] // "Unknown")'
Writing multi-line expressions (Workshop Example)
You can make long jq filters in the Workshop more readable by splitting them over multiple lines. To do this, use the >-
YAML operator that replaces newlines with spaces, and strips off the last new line (Read more about multi-line YAML strings)
For example, to clean up the filter in the "Looking up a value in a pre-defined map" example above:
filter: >-
.mapping as $m |
.items |
map($m[.])
Multi-line expressions are also helpful for constructing objects with many properties. (Note that JQ requires parentheses around the object's values):
filter: >-
{
fooCheck: (.foo > 42),
barCheck: (.bar | length > 0),
}