pyspark.sql.functions.variant_array_append#

pyspark.sql.functions.variant_array_append(v, path, value)[source]#

Appends a value to the array in a variant at the given JSONPath location. Returns the variant unchanged if a path key or index is absent. Throws an error if a path segment hits a value of an incompatible type or the target is not an array. Returns NULL if any argument is NULL.

Added in version 4.3.0.

Parameters:

vColumn or str: a variant column or column name A column that evaluates to a variant.
pathColumn or str: the JSONPath target array. A str is a literal path; a Column supplies the path at runtime. A valid path should start with $ and is followed by zero or A column that evaluates to a string. more segments like [123], .name, [‘name’], or [“name”].
valueColumn or str: the value to append. Any expression castable to variant.

Returns:

Column: a variant column with value appended to the array at path Returns a column that evaluates to a variant.

Examples

>>> from pyspark.sql.functions import lit, parse_json, to_json, variant_array_append
>>> df = spark.createDataFrame([{
...     'json': '''[[1, 2], 5]''',
...     'path': '$[0]'
... }])
>>> v = parse_json(df.json)
>>> df.select(to_json(variant_array_append(v, "$", lit(3))).alias("r")).collect()
[Row(r='[[1,2],5,3]')]
>>> df.select(to_json(variant_array_append(v, "$[5]", lit(3))).alias("r")).collect()
[Row(r='[[1,2],5]')]
>>> df.select(to_json(variant_array_append(v, df.path, lit(9))).alias("r")).collect()
[Row(r='[[1,2,9],5]')]
>>> nested = variant_array_append(v, "$", parse_json(lit('[4, 5]')))
>>> df.select(to_json(nested).alias("r")).collect()
[Row(r='[[1,2],5,[4,5]]')]
>>> df.select(variant_array_append(v, "$", lit(None)).alias("r")).collect()
[Row(r=None)]