pyspark.sql.functions.variant_delete#

pyspark.sql.functions.variant_delete(v, *paths)[source]#

Removes fields or array elements from a variant at the given JSONPath locations. Multiple paths are applied left to right. Returns NULL if v is NULL; NULL paths are skipped.

Added in version 4.3.0.

Parameters:

vColumn or str: a variant column or column name A column that evaluates to a variant.
pathsColumn or str: one or more JSONPath deletion targets. A str is a literal path; a Column supplies the path at runtime. A valid path should start with $ and is followed by one or more segments like [123], .name, [‘name’], or [“name”]. The root path $ is not A column that evaluates to a string. allowed.

Returns:

Column: a variant column with the specified paths removed Returns a column that evaluates to a variant.

Examples

>>> from pyspark.sql.functions import lit, parse_json, to_json, variant_delete
>>> df = spark.createDataFrame([{
...     'json': '''{ "a" : 1, "b" : 2, "c" : 3, "items" : [1, 2, 3] }''',
...     'path': '$.a'
... }])
>>> v = parse_json(df.json)
>>> df.select(to_json(variant_delete(v, lit(None), "$.a", "$.c")).alias("r")).collect()
[Row(r='{"b":2,"items":[1,2,3]}')]
>>> df.select(to_json(variant_delete(v, "$.missing")).alias("r")).collect()
[Row(r='{"a":1,"b":2,"c":3,"items":[1,2,3]}')]
>>> df.select(to_json(variant_delete(v, df.path)).alias("r")).collect()
[Row(r='{"b":2,"c":3,"items":[1,2,3]}')]
>>> df.select(to_json(variant_delete(v, "$.items[0]", "$.items[0]")).alias("r")).collect()
[Row(r='{"a":1,"b":2,"c":3,"items":[3]}')]
>>> df.select(variant_delete(lit(None), "$.a").alias("r")).collect()
[Row(r=None)]