pyspark.sql.functions.tuple_sketch_agg_integer#

pyspark.sql.functions.tuple_sketch_agg_integer(key, summary, lgNomEntries=None, mode=None)[source]#

Aggregate function: returns the compact binary representation of the Datasketches TupleSketch with integer summaries built from the key and summary columns.

Added in version 4.2.0.

Parameters:

keyColumn or column name: The column containing key values. A column that evaluates to an array, binary, double, float, integer, long, or string.
summaryColumn or column name: The column containing integer summary values. A column that evaluates to an integer.
lgNomEntriesColumn or int, optional: The log-base-2 of nominal entries (must be between 4 and 26, defaults to 12) A column that evaluates to an integer.
modeColumn or str, optional: The summary mode: “sum” (default), “min”, “max”, or “alwaysone”

Returns:

Column: The binary representation of the TupleSketch.

See also

pyspark.sql.functions.tuple_sketch_estimate_integer()
pyspark.sql.functions.tuple_sketch_summary_integer()
pyspark.sql.functions.tuple_union_agg_integer()

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([(1, 10), (2, 20), (2, 30)], ["key", "value"])
>>> df.agg(sf.tuple_sketch_estimate_integer(
...     sf.tuple_sketch_agg_integer("key", "value"))).show()
+----------------------------------------------------------------------------+
|tuple_sketch_estimate_integer(tuple_sketch_agg_integer(key, value, 12, sum))|
+----------------------------------------------------------------------------+
|                                                                         2.0|
+----------------------------------------------------------------------------+