Using JQ in PixieBrix

What is jq?

jq is a powerful expression language for filtering and transforming structured JSON data. It's available for the command line and a variety of programming languages, including Java, Python, and Javascript.

Helpful Resources

• jqplay: A playground for jq 1.6

• jq Manual: has runnable examples of each operator

• jq Homepage

• jq tag on StackOverflow

Brick Arguments

The jq brick takes two input arguments:

• data: an object, array, or single value which is passed to jq

• filter: the jq "filter", which controls how to filter/transform the data

Brick Argument Examples

Scenario #1: passing a variable for data

You can pass a variable or template directly to data:

id: "@pixiebrix/jq"
config:
  # @apiResult = {items: []}
  data: "@apiResult"
  filter: ".items"

Scenario #2: passing multiple variables to data

You can also pass multiple variables by passing a dictionary to data, including using template syntax:

id: "@pixiebrix/jq"
config:
  data: 
    # @apiResult = {items: [{text: "foo"}]}
    result: "@apiResult"
    # @input = {query: "foo"}
    query: "{{ @input.query }}!"
  filter: ".query as $query | [.result.items[] | select(.text == $query)]"

Common Pitfalls

• jq only has access to the data you passed in via the data argument. You should not use @variable syntax in the jq filter

• jq uses a . to refer to its input (the data you passed to the brick)

• jq can return multiple results. However, PixieBrix expects a single result from jq. Therefore, you must wrap an expression that returns multiple results in Array construction [] operator

Filter Examples

Creating a new object from the input

▶️ View on jqplay

{foo: 42, bar: .}

Parsing and summing values

▶️ View on jqplay

.transactions | map(.amount | capture("(?<val>\\\\d+) USD") | .val | tonumber) | add

Filtering an array

▶️ View on jqplay

Assign the query to a variable so you can use it in the select clause.

NOTE: jq's select operator returns multiple results. Wrap the expression in an Array Construction [] operator to return the results as a single array

.query as $query | [.items[] | select(.text == $query)]

Looking up a value in a pre-defined map (Workshop Example)

▶️ View on jqplay

Pass the pre-defined map as an input to the jq brick. Then, assign the map to a variable (e.g., $m) so you can lookup up the value later in the filter:

data:
  # ["A", "B", "C"]
  items: "@items"
  mapping:
    A: Alpha
    B: Bravo
    C: Charlie
filter: ".mapping as $m | .items | map($m[.])"

You can use jq's // defaulting operator to handle elements that aren't in the map. E.g.,

filter: '.mapping as $m | .items | map($m[.] // "Unknown")'

Writing multi-line expressions (Workshop Example)

You can make long jq filters in the Workshop more readable by splitting them over multiple lines. To split a filter over multiple lines, use the >- YAML operator that replaces newlines with spaces, and strips off the last new line (Read more about multi-line YAML strings)

For example, to clean up the filter in the "Looking up a value in a pre-defined map" example above:

filter: >-
  .mapping as $m |
  .items |
  map($m[.])

Multi-line expressions are also helpful for constructing objects with many properties. (Note that JQ requires parentheses around the object's values):

filter: >-
  {
    fooCheck: (.foo > 42),
    barCheck: (.bar | length > 0),
  }