LDAModel

class pyspark.ml.clustering.LDAModel(java_model: Optional[JavaObject] = None)

Latent Dirichlet Allocation (LDA) model. This abstraction permits for different underlying representations, including local and distributed data structures.

New in version 2.0.0.

Methods

clear(param)	Clears a param from the param map if it has been explicitly set.
copy([extra])	Creates a copy of this instance with the same uid and some extra params.
describeTopics([maxTermsPerTopic])	Return the topics described by their top-weighted terms.
estimatedDocConcentration()	Value for LDA.docConcentration estimated from data.
explainParam(param)	Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
explainParams()	Returns the documentation of all params with their optionally default values and user-supplied values.
extractParamMap([extra])	Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
getCheckpointInterval()	Gets the value of checkpointInterval or its default value.
getDocConcentration()	Gets the value of docConcentration or its default value.
getFeaturesCol()	Gets the value of featuresCol or its default value.
getK()	Gets the value of k or its default value.
getKeepLastCheckpoint()	Gets the value of keepLastCheckpoint or its default value.
getLearningDecay()	Gets the value of learningDecay or its default value.
getLearningOffset()	Gets the value of learningOffset or its default value.
getMaxIter()	Gets the value of maxIter or its default value.
getOptimizeDocConcentration()	Gets the value of optimizeDocConcentration or its default value.
getOptimizer()	Gets the value of optimizer or its default value.
getOrDefault(param)	Gets the value of a param in the user-supplied param map or its default value.
getParam(paramName)	Gets a param by its name.
getSeed()	Gets the value of seed or its default value.
getSubsamplingRate()	Gets the value of subsamplingRate or its default value.
getTopicConcentration()	Gets the value of topicConcentration or its default value.
getTopicDistributionCol()	Gets the value of topicDistributionCol or its default value.
hasDefault(param)	Checks whether a param has a default value.
hasParam(paramName)	Tests whether this instance contains a param with a given (string) name.
isDefined(param)	Checks whether a param is explicitly set by user or has a default value.
isDistributed()	Indicates whether this instance is of type DistributedLDAModel
isSet(param)	Checks whether a param is explicitly set by user.
logLikelihood(dataset)	Calculates a lower bound on the log likelihood of the entire corpus.
logPerplexity(dataset)	Calculate an upper bound on perplexity.
set(param, value)	Sets a parameter in the embedded param map.
setFeaturesCol(value)	Sets the value of featuresCol.
setSeed(value)	Sets the value of seed.
setTopicDistributionCol(value)	Sets the value of topicDistributionCol.
topicsMatrix()	Inferred topics, where each topic is represented by a distribution over terms.
transform(dataset[, params])	Transforms the input dataset with optional parameters.
vocabSize()	Vocabulary size (number of terms or words in the vocabulary)

Attributes

checkpointInterval
docConcentration
featuresCol
k
keepLastCheckpoint
learningDecay
learningOffset
maxIter
optimizeDocConcentration
optimizer
params	Returns all params ordered by name.
seed
subsamplingRate
topicConcentration
topicDistributionCol

Methods Documentation

clear(param: pyspark.ml.param.Param) → None

Clears a param from the param map if it has been explicitly set.

copy(extra: Optional[ParamMap] = None) → JP

Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.

Parameters

extradict, optional

Extra parameters to copy to the new instance

Returns

JavaParams

Copy of this instance

describeTopics(maxTermsPerTopic: int = 10) → pyspark.sql.dataframe.DataFrame

Return the topics described by their top-weighted terms.

New in version 2.0.0.

estimatedDocConcentration() → pyspark.ml.linalg.Vector

Value for LDA.docConcentration estimated from data. If Online LDA was used and LDA.optimizeDocConcentration was set to false, then this returns the fixed (given) value for the LDA.docConcentration parameter.

New in version 2.0.0.

explainParam(param: Union[str, pyspark.ml.param.Param]) → str

Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str

Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: Optional[ParamMap] = None) → ParamMap

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters

extradict, optional

extra param values

Returns

dict

merged param map

getCheckpointInterval() → int

Gets the value of checkpointInterval or its default value.

getDocConcentration() → List[float]

Gets the value of docConcentration or its default value.

New in version 2.0.0.

getFeaturesCol() → str

Gets the value of featuresCol or its default value.

getK() → int

Gets the value of k or its default value.

New in version 2.0.0.

getKeepLastCheckpoint() → bool

Gets the value of keepLastCheckpoint or its default value.

New in version 2.0.0.

getLearningDecay() → float

Gets the value of learningDecay or its default value.

New in version 2.0.0.

getLearningOffset() → float

Gets the value of learningOffset or its default value.

New in version 2.0.0.

getMaxIter() → int

Gets the value of maxIter or its default value.

getOptimizeDocConcentration() → bool

Gets the value of optimizeDocConcentration or its default value.

New in version 2.0.0.

getOptimizer() → str

Gets the value of optimizer or its default value.

New in version 2.0.0.

getOrDefault(param: Union[str, pyspark.ml.param.Param[T]]) → Union[Any, T]

Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getParam(paramName: str) → pyspark.ml.param.Param

Gets a param by its name.

getSeed() → int

Gets the value of seed or its default value.

getSubsamplingRate() → float

Gets the value of subsamplingRate or its default value.

New in version 2.0.0.

getTopicConcentration() → float

Gets the value of topicConcentration or its default value.

New in version 2.0.0.

getTopicDistributionCol() → str

Gets the value of topicDistributionCol or its default value.

New in version 2.0.0.

hasDefault(param: Union[str, pyspark.ml.param.Param[Any]]) → bool

Checks whether a param has a default value.

hasParam(paramName: str) → bool

Tests whether this instance contains a param with a given (string) name.

isDefined(param: Union[str, pyspark.ml.param.Param[Any]]) → bool

Checks whether a param is explicitly set by user or has a default value.

isDistributed() → bool

Indicates whether this instance is of type DistributedLDAModel

New in version 2.0.0.

isSet(param: Union[str, pyspark.ml.param.Param[Any]]) → bool

Checks whether a param is explicitly set by user.

logLikelihood(dataset: pyspark.sql.dataframe.DataFrame) → float

Calculates a lower bound on the log likelihood of the entire corpus. See Equation (16) in the Online LDA paper (Hoffman et al., 2010).

Warning

If this model is an instance of DistributedLDAModel (produced when optimizer is set to “em”), this involves collecting a large topicsMatrix() to the driver. This implementation may be changed in the future.

New in version 2.0.0.

logPerplexity(dataset: pyspark.sql.dataframe.DataFrame) → float

Calculate an upper bound on perplexity. (Lower is better.) See Equation (16) in the Online LDA paper (Hoffman et al., 2010).

Warning

If this model is an instance of DistributedLDAModel (produced when optimizer is set to “em”), this involves collecting a large topicsMatrix() to the driver. This implementation may be changed in the future.

New in version 2.0.0.

set(param: pyspark.ml.param.Param, value: Any) → None

Sets a parameter in the embedded param map.

setFeaturesCol(value: str) → M

Sets the value of featuresCol.

New in version 3.0.0.

setSeed(value: int) → M

Sets the value of seed.

New in version 3.0.0.

setTopicDistributionCol(value: str) → M

Sets the value of topicDistributionCol.

New in version 3.0.0.

topicsMatrix() → pyspark.ml.linalg.Matrix

Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.

Warning

If this model is actually a DistributedLDAModel instance produced by the Expectation-Maximization (“em”) optimizer, then this method could involve collecting a large amount of data to the driver (on the order of vocabSize x k).

New in version 2.0.0.

transform(dataset: pyspark.sql.dataframe.DataFrame, params: Optional[ParamMap] = None) → pyspark.sql.dataframe.DataFrame

Transforms the input dataset with optional parameters.

New in version 1.3.0.

Parameters

datasetpyspark.sql.DataFrame

input dataset

paramsdict, optional

an optional param map that overrides embedded params.

Returns

pyspark.sql.DataFrame

transformed dataset

vocabSize() → int

Vocabulary size (number of terms or words in the vocabulary)

New in version 2.0.0.

Attributes Documentation

checkpointInterval = Param(parent='undefined', name='checkpointInterval', doc='set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext.')

docConcentration: pyspark.ml.param.Param[List[float]] = Param(parent='undefined', name='docConcentration', doc='Concentration parameter (commonly named "alpha") for the prior placed on documents\' distributions over topics ("theta").')

featuresCol = Param(parent='undefined', name='featuresCol', doc='features column name.')

k: pyspark.ml.param.Param[int] = Param(parent='undefined', name='k', doc='The number of topics (clusters) to infer. Must be > 1.')

keepLastCheckpoint: pyspark.ml.param.Param[bool] = Param(parent='undefined', name='keepLastCheckpoint', doc='(For EM optimizer) If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care.')

learningDecay: pyspark.ml.param.Param[float] = Param(parent='undefined', name='learningDecay', doc='Learning rate, set as anexponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence.')

learningOffset: pyspark.ml.param.Param[float] = Param(parent='undefined', name='learningOffset', doc='A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less')

maxIter = Param(parent='undefined', name='maxIter', doc='max number of iterations (>= 0).')

optimizeDocConcentration: pyspark.ml.param.Param[bool] = Param(parent='undefined', name='optimizeDocConcentration', doc='Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training.')

optimizer: pyspark.ml.param.Param[str] = Param(parent='undefined', name='optimizer', doc='Optimizer or inference algorithm used to estimate the LDA model. Supported: online, em')

params

Returns all params ordered by name. The default implementation uses dir() to get all attributes of type Param.

seed = Param(parent='undefined', name='seed', doc='random seed.')

subsamplingRate: pyspark.ml.param.Param[float] = Param(parent='undefined', name='subsamplingRate', doc='Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1].')

topicConcentration: pyspark.ml.param.Param[float] = Param(parent='undefined', name='topicConcentration', doc='Concentration parameter (commonly named "beta" or "eta") for the prior placed on topic\' distributions over terms.')

topicDistributionCol: pyspark.ml.param.Param[str] = Param(parent='undefined', name='topicDistributionCol', doc='Output column with estimates of the topic mixture distribution for each document (often called "theta" in the literature). Returns a vector of zeros for an empty document.')

LDA LocalLDAModel