pyspark.sql.functions.theta_sketch_agg#

pyspark.sql.functions.theta_sketch_agg(col, lgNomEntries=None)[source]#

Aggregate function: returns the compact binary representation of the Datasketches ThetaSketch with the values in the input column configured with lgNomEntries nominal entries.

Added in version 4.1.0.

Parameters:

colColumn or column name: A column that evaluates to an array, binary, double, float, integer, long, or string.
lgNomEntriesColumn or int, optional: The log-base-2 of nominal entries, where nominal entries is the size of the sketch (must be between 4 and 26, defaults to 12). A column that evaluates to an integer.

Returns:

Column: The binary representation of the ThetaSketch.

See also

pyspark.sql.functions.theta_union()
pyspark.sql.functions.theta_intersection()
pyspark.sql.functions.theta_difference()
pyspark.sql.functions.theta_union_agg()
pyspark.sql.functions.theta_intersection_agg()
pyspark.sql.functions.theta_sketch_estimate()

Examples

>>> from pyspark.sql import functions as sf
>>> df = spark.createDataFrame([1,2,2,3], "INT")
>>> df.agg(sf.theta_sketch_estimate(sf.theta_sketch_agg("value"))).show()
+--------------------------------------------------+
|theta_sketch_estimate(theta_sketch_agg(value, 12))|
+--------------------------------------------------+
|                                                 3|
+--------------------------------------------------+

>>> df.agg(sf.theta_sketch_estimate(sf.theta_sketch_agg("value", 15))).show()
+--------------------------------------------------+
|theta_sketch_estimate(theta_sketch_agg(value, 15))|
+--------------------------------------------------+
|                                                 3|
+--------------------------------------------------+