pyspark.sql.functions.variant_set#
- pyspark.sql.functions.variant_set(v, path, value, create_if_missing=True)[source]#
Sets or upserts a value in a variant at the given JSONPath location. An existing object field or array element at the target is replaced. A missing field, array index, or intermediate path is created, unless create_if_missing is false, in which case the variant is left unchanged. Throws an error if a path segment hits a value of an incompatible type. Returns NULL if any argument is NULL.
Added in version 4.3.0.
- Parameters:
- v
Columnor str a variant column or column name A column that evaluates to a variant.
- path
Columnor str the JSONPath set target. A str is a literal path; a
Columnsupplies the path at runtime. A valid path should start with $ and is followed by one or more A column that evaluates to a string. segments like [123], .name, [‘name’], or [“name”]. The root path $ is not allowed.- value
Columnor str the value to set. Any expression castable to variant.
- create_if_missingbool, optional
whether to create missing keys or out-of-range array indices (default True). A column that evaluates to a boolean. Must be a constant.
- v
- Returns:
Columna variant column with value set at path Returns a column that evaluates to a variant.
Examples
>>> from pyspark.sql.functions import lit, parse_json, to_json, variant_set >>> df = spark.createDataFrame([{'json': '''{"a": 1, "arr": [1, 2, 3]}'''}]) >>> v = parse_json(df.json) >>> df.select(to_json(variant_set(v, "$.a", lit(9))).alias("r")).collect() [Row(r='{"a":9,"arr":[1,2,3]}')] >>> df.select(to_json(variant_set(v, "$.b", lit(2))).alias("r")).collect() [Row(r='{"a":1,"arr":[1,2,3],"b":2}')] >>> df.select(to_json(variant_set(v, "$.arr[1]", lit(9))).alias("r")).collect() [Row(r='{"a":1,"arr":[1,9,3]}')] >>> df.select(to_json(variant_set(v, "$.b", lit(2), False)).alias("r")).collect() [Row(r='{"a":1,"arr":[1,2,3]}')] >>> df.select(to_json(variant_set(v, "$.a", parse_json(lit("null")))).alias("r")).collect() [Row(r='{"a":null,"arr":[1,2,3]}')] >>> df.select(to_json(variant_set(v, "$.a", lit(None))).alias("r")).collect() [Row(r=None)]