pyspark.sql.functions.tuple_union_agg_integer#

pyspark.sql.functions.tuple_union_agg_integer(col, lgNomEntries=None, mode=None)[source]#

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch that is the union of the integer TupleSketch objects in the input column.

Added in version 4.2.0.

Parameters:

colColumn or column name: The column containing binary TupleSketch representations. A column that evaluates to a binary.
lgNomEntriesColumn or int, optional: The log-base-2 of nominal entries (must be between 4 and 26, defaults to 12) A column that evaluates to an integer.
modeColumn or str, optional: The summary mode: “sum” (default), “min”, “max”, or “alwaysone”

Returns:

Column: The binary representation of the merged TupleSketch.

See also

pyspark.sql.functions.tuple_sketch_agg_integer()
pyspark.sql.functions.tuple_union_integer()

Examples

>>> from pyspark.sql import functions as sf
>>> df1 = spark.createDataFrame([(1, 10), (2, 20)], ["key", "value"])
>>> df1 = df1.agg(sf.tuple_sketch_agg_integer("key", "value").alias("sketch"))
>>> df2 = spark.createDataFrame([(3, 30), (4, 40)], ["key", "value"])
>>> df2 = df2.agg(sf.tuple_sketch_agg_integer("key", "value").alias("sketch"))
>>> df3 = df1.union(df2)
>>> df3.agg(sf.tuple_sketch_estimate_integer(sf.tuple_union_agg_integer("sketch"))).show()
+-----------------------------------------------------------------------+
|tuple_sketch_estimate_integer(tuple_union_agg_integer(sketch, 12, sum))|
+-----------------------------------------------------------------------+
|                                                                    4.0|
+-----------------------------------------------------------------------+