📖
PixieBrix Documentation
🔀

Working with jq

What is jq?

jq is a powerful expression language for filtering and transforming structured JSON data. It's available for the command line and a variety of programming languages, including Java, Python, and Javascript.

Helpful Resources

Brick Arguments

The jq brick takes two input arguments:

  • data: an object, array, or single value which is passed to jq
  • filter: the jq "filter", which controls how to filter/transform the data

Brick Argument Examples

Scenario #1: passing a variable for data

You can pass a variable or template directly to data:

id: "@pixiebrix/jq"
config:
  # @apiResult = {items: []}
  data: "@apiResult"
  filter: ".items"

Scenario #2: passing multiple variables to data

You can also pass multiple variables by passing a dictionary to data, including using template syntax:

id: "@pixiebrix/jq"
config:
  data: 
    # @apiResult = {items: [{text: "foo"}]}
    result: "@apiResult"
    # @input = {query: "foo"}
    query: "{{ @input.query }}!"
  filter: ".query as $query | [.result.items[] | select(.text == $query)]"

Common Pitfalls

  • jq only has access to the data you passed in via the data argument. You should not use @variable syntax in the jq filter
  • jq uses a . to refer to its input (the data you passed to the brick)
  • jq can return multiple results. However, PixieBrix expects a single result from jq. Therefore, you must wrap an expression that returns multiple results in Array construction [] operator

Filter Examples

Creating a new object from the input

▶️  View on jqplay

{foo: 42, bar: .}

Parsing and summing values

▶️  View on jqplay

.transactions | map(.amount | capture("(?<val>\\d+) USD") | .val | tonumber) | add

Filtering an array

▶️  View on jqplay

Assign the query to a variable so you can use it in the select clause.

NOTE: jq's select operator returns multiple results. Wrap the expression in an Array Construction [] operator to return the results as a single array

.query as $query | [.items[] | select(.text == $query)]

Looking up a value in a pre-defined map (Workshop Example)

▶️  View on jqplay

Pass the pre-defined map as an input to the jq brick. Then, assign the map to a variable (e.g., $m) so you can lookup up the value later in the filter:

data:
  # ["A", "B", "C"]
	items: "@items"
  mapping:
		A: Alpha
    B: Bravo
    C: Charlie
filter: ".mapping as $m | .items | map($m[.])"

You can use jq's // defaulting operator to handle elements that aren't in the map. E.g., 

filter: '.mapping as $m | .items | map($m[.] // "Unknown")'

Writing multi-line expressions (Workshop Example)

You can make long jq filters in the Workshop more readable by splitting them over multiple lines. To do this, use the >- YAML operator that replaces newlines with spaces, and strips off the last new line (Read more about multi-line YAML strings)

For example, to clean up the filter in the "Looking up a value in a pre-defined map" example above:

filter: >-
  .mapping as $m |
  .items |
  map($m[.])

Multi-line expressions are also helpful for constructing objects with many properties. (Note that JQ requires parentheses around the object's values):

filter: >-
  {
		fooCheck: (.foo > 42),
		barCheck: (.bar | length > 0),
  }

Twitter