datafusion-variant

Variant support for Apache DataFusion, with functions and SQL syntax for querying and manipulating semi-structured data.

Variant values can contain primitives, objects, and arrays without requiring a fixed schema for their contents.

Project status

This crate is under active development. The compatibility target is Apache Spark 4.2.0. Compatibility is incomplete, and function names, signatures, and behavior may change.

SQL VARIANT type registration, basic : field access, and casts of getters through CAST, TRY_CAST, and :: are available.

See #69 for the compatibility roadmap, integration work, and benchmarking plans.

Getting started

Register the functions you need with a DataFusion SessionContext. Use register_variant_get_functions for SQL type hints and getter casts, and register VariantExprPlanner to enable : field access. Install VariantTypePlanner when constructing the session for SQL VARIANT syntax:

use std::sync::Arc;

use datafusion::error::Result;
use datafusion::execution::{FunctionRegistry, SessionStateBuilder};
use datafusion::logical_expr::ScalarUDF;
use datafusion::prelude::SessionContext;
use datafusion_variant::{
    ParseJsonUdf, VariantExprPlanner, VariantToJsonUdf, VariantTypePlanner,
    register_variant_get_functions,
};

#[tokio::main]
async fn main() -> Result<()> {
    let state = SessionStateBuilder::new()
        .with_default_features()
        .with_type_planner(Arc::new(VariantTypePlanner))
        .build();
    let mut ctx = SessionContext::new_with_state(state);

    ctx.register_udf(ScalarUDF::new_from_impl(ParseJsonUdf::default()));
    ctx.register_udf(ScalarUDF::new_from_impl(VariantToJsonUdf::default()));
    register_variant_get_functions(&mut ctx)?;
    ctx.register_expr_planner(Arc::new(VariantExprPlanner))?;

    ctx.sql(
        r#"
        SELECT to_json(
            parse_json('{"name":"Alice","age":30}'):name
        ) AS name
        "#,
    )
    .await?
    .show()
    .await?;

    Ok(())
}

This returns the JSON string "Alice".

Use compatible DataFusion and Arrow dependencies when embedding the crate. See Cargo.toml for the current versions and DataFusion revision. The example also requires Tokio with its macros and rt-multi-thread features.

Functionality

The current API includes:

Area	Functions
JSON conversion	parse_json, try_parse_json, to_json, variant_get_json
Type conversion	cast_to_variant
Field and path access	variant_get, try_variant_get, variant_contains
Null inspection	is_variant_null
Schema inference	schema_of_variant, schema_of_variant_agg
Objects	variant_object_construct, variant_object_keys, variant_object_insert, variant_object_delete
Arrays	variant_list_construct, variant_list_insert, variant_list_delete
Utilities	variant_normalize, variant_pretty

try_parse_json accepts the same inputs as parse_json, but returns SQL NULL for invalid JSON values. Valid JSON null remains a Variant null. Register it with TryParseJsonUdf::default(). Both functions currently share Arrow's JSON parsing limitations, including decimal precision and duplicate-key handling.

to_json serializes Variant values and is registered with VariantToJsonUdf::default(). Its SQL name replaces variant_to_json to match Spark; callers using the old SQL name must update their queries.

schema_of_variant(v) returns a SQL schema for each value, for example OBJECT<a: BIGINT, b: ARRAY<STRING>>. It merges types within arrays, preserves SQL NULL, and reports VOID for Variant null. It follows Spark's schema_of_variant inference rules. Arrow's JSON parser produces DOUBLE for fractional JSON numbers; use cast_to_variant with SQL decimals to retain decimal precision and scale. Register it with SchemaOfVariantUdf::default().

schema_of_variant_agg(v) merges schemas across rows, skipping SQL NULLs. Empty and all-null groups return VOID. It shares the scalar function's inference rules. Register it with ctx.register_udaf(AggregateUDF::new_from_impl(SchemaOfVariantAggUdf::default())).

variant_get_json(v, 'a') extracts a value and serializes it as JSON, including quotes around strings. It retains a direct extraction path that avoids the intermediate Variant array in to_json(variant_get(v, 'a')).

variant_get and try_variant_get accept an optional SQL type string literal, such as BIGINT, VARCHAR, DECIMAL(12,2), or ARRAY<INT>. Types follow DataFusion SQL semantics exclusively: INT8 (including Int8) means BIGINT; Arrow-only names such as Int64 and Utf8View are rejected. variant_get raises an error on failed conversions; try_variant_get returns NULL for those values. Both return NULL for missing paths; invalid paths and type hints still raise errors. Use quoted bracket paths for literal field names, such as variant_get(v, '["http.status"]') for the key http.status. See literal-key tests for examples and skipped cases awaiting an upstream Arrow path-parser fix.

These expressions share the same extraction/conversion kernel:

variant_get(v, 'price', 'DECIMAL(12,2)')
CAST(variant_get(v, 'price') AS DECIMAL(12,2))
v:price::DECIMAL(12,2)

TRY_CAST uses the try-getter's conversion policy. Casts of already typed getters retain the intermediate conversion. With the current parser, group colon access on the right of arithmetic: v:a::INT + (v:b::INT).

Type strings use DataFusion's SQL cast planner during planning. The registration helper follows session type settings, including SET / RESET.

VARIANT resolves to a struct with metadata and value binary-view fields and the arrow.parquet.variant extension metadata. Getter hints such as variant_get(v, 'price', 'VARIANT') and getter casts such as v:price::VARIANT use Arrow's native Variant target conversion. Use cast_to_variant and typed getters to convert to/from Variant columns; general SQL casts of those columns are not yet supported. Nested Variant getter targets are also unsupported.

See the Spark field-access tests for remaining conversion and syntax gaps.

Examples

The CLI example loads a sample Bluesky JSON dataset into an in-memory table named bsky and registers Variant functions and : field access.

From the repository root, download the sample data and choose option 1:

bash download_data.sh

Start the CLI:

cargo run --example cli

Queries must end with a semicolon:

SELECT to_json(parse_json(json_data))
FROM bsky
LIMIT 5;

Enter quit or q to exit. The example loads data/bluesky/file_0001.json.gz into memory before accepting queries.

See examples for the source code.

Benchmarks

Run the extraction benchmarks:

cargo bench --bench variant_get

The suite has four groups, each using 8,192 rows. The first three use literal paths:

• output_types: compares Variant output and explicit type hints for integer, float, boolean, and string values at path a in unshredded objects.
• storage_layout: extracts Variant output from identical two-field objects in unshredded, fully shredded, and partially shredded layouts. The partial layout shreds only sibling field b, leaving a in binary storage.
• path_traversal: extracts the same integer as Variant from the same unshredded documents via top, obj.b.c, arr[1][1], and mix[1].b. The nested paths each have three steps; top is a one-step control. Timing includes parsing the path once per batch, per-row traversal, and output construction.
• path_columns: compares a literal a, a UTF-8 column repeating a, and a UTF-8 column alternating a/b over the same unshredded objects containing positive integers at a and their negatives at b, with Variant output. These call the UDF directly, exercising scalar versus array path dispatch without SQL optimization. Path columns are constructed outside timing.

Run one group by filtering its name:

cargo bench --bench variant_get -- output_types
cargo bench --bench variant_get -- storage_layout
cargo bench --bench variant_get -- path_traversal
cargo bench --bench variant_get -- path_columns

See the benchmark source for the workloads.

Run serialization benchmarks with cargo bench --bench to_json. They cover scalar SQL NULL, plus scalar inputs and 8,192-row arrays of integers, Variant nulls, escaped strings, and nested values, plus arrays with 25% and 100% SQL NULLs. The storage_layout groups compare identical integers and escaped strings in unshredded and typed-column storage, for both scalar and column inputs. Inputs are constructed before timing; measurements include UDF invocation, argument cloning, and output allocation/drop. Shredded objects are excluded until their reconstruction issues are fixed (#71). See the source.

Run schema inference benchmarks with cargo bench --bench schema_of_variant. They cover scalar and array inputs, aggregate updates, state serialization, and partial-state merging, including nulls, nested values, widening, and shredded integers. Input construction and validation are outside timing. See the source for workloads and timing boundaries.

To compare a change, save a baseline before making it, then rerun afterward:

# Before the change
cargo bench --bench variant_get -- --save-baseline before

# After the change
cargo bench --bench variant_get -- --baseline before

Use the same machine and build settings, and preserve target/criterion between runs. Include relevant results when proposing performance changes.

Broader benchmark coverage is tracked in #19.

For realistic Parquet workloads, JSONBench prepares the original Bluesky data as unshredded, partially shredded, and query-field-shredded Variant. Its default run executes all five adapted queries against all three layouts, reusing prepared fixtures:

cargo run --release --example jsonbench -- prepare
cargo run --release --example jsonbench -- run

Contributing

Contributions are welcome, including compatibility fixes, regression tests, benchmarks, documentation, and integration examples.

Use a current stable Rust toolchain with rustfmt and clippy, and install the Protocol Buffers compiler (protoc).

Run the development checks from the repository root:

cargo test
cargo fmt --all -- --check
cargo clippy -- -D warnings

For Spark compatibility changes, reference the upstream behavior and add regression coverage in the SQL tests.

The issue tracker contains individual tasks and discussions. Overall direction is tracked in #69.

References

• Apache Spark 4.2.0 documentation
• Parquet Variant encoding
• Arrow canonical extension types
• Custom types using metadata in DataFusion
• Iceberg Variant proposal — design background

License

Licensed under the Apache License, Version 2.0.