DistributedLDAModel¶
-
class
pyspark.ml.clustering.
DistributedLDAModel
(java_model: Optional[JavaObject] = None)[source]¶ Distributed model fitted by
LDA
. This type of model is currently only produced by Expectation-Maximization (EM).This model stores the inferred topics, the full training dataset, and the topic distribution for each training document.
New in version 2.0.0.
Methods
clear
(param)Clears a param from the param map if it has been explicitly set.
copy
([extra])Creates a copy of this instance with the same uid and some extra params.
describeTopics
([maxTermsPerTopic])Return the topics described by their top-weighted terms.
Value for
LDA.docConcentration
estimated from data.explainParam
(param)Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
Returns the documentation of all params with their optionally default values and user-supplied values.
extractParamMap
([extra])Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
If using checkpointing and
LDA.keepLastCheckpoint
is set to true, then there may be saved checkpoint files.Gets the value of checkpointInterval or its default value.
Gets the value of
docConcentration
or its default value.Gets the value of featuresCol or its default value.
getK
()Gets the value of
k
or its default value.Gets the value of
keepLastCheckpoint
or its default value.Gets the value of
learningDecay
or its default value.Gets the value of
learningOffset
or its default value.Gets the value of maxIter or its default value.
Gets the value of
optimizeDocConcentration
or its default value.Gets the value of
optimizer
or its default value.getOrDefault
(param)Gets the value of a param in the user-supplied param map or its default value.
getParam
(paramName)Gets a param by its name.
getSeed
()Gets the value of seed or its default value.
Gets the value of
subsamplingRate
or its default value.Gets the value of
topicConcentration
or its default value.Gets the value of
topicDistributionCol
or its default value.hasDefault
(param)Checks whether a param has a default value.
hasParam
(paramName)Tests whether this instance contains a param with a given (string) name.
isDefined
(param)Checks whether a param is explicitly set by user or has a default value.
Indicates whether this instance is of type DistributedLDAModel
isSet
(param)Checks whether a param is explicitly set by user.
load
(path)Reads an ML instance from the input path, a shortcut of read().load(path).
logLikelihood
(dataset)Calculates a lower bound on the log likelihood of the entire corpus.
logPerplexity
(dataset)Calculate an upper bound on perplexity.
logPrior
()Log probability of the current parameter estimate: log P(topics, topic distributions for docs | alpha, eta)
read
()Returns an MLReader instance for this class.
save
(path)Save this ML instance to the given path, a shortcut of ‘write().save(path)’.
set
(param, value)Sets a parameter in the embedded param map.
setFeaturesCol
(value)Sets the value of
featuresCol
.setSeed
(value)Sets the value of
seed
.setTopicDistributionCol
(value)Sets the value of
topicDistributionCol
.toLocal
()Convert this distributed model to a local representation.
Inferred topics, where each topic is represented by a distribution over terms.
Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, Dirichlet hyperparameters)
transform
(dataset[, params])Transforms the input dataset with optional parameters.
Vocabulary size (number of terms or words in the vocabulary)
write
()Returns an MLWriter instance for this ML instance.
Attributes
Returns all params ordered by name.
Methods Documentation
-
clear
(param: pyspark.ml.param.Param) → None¶ Clears a param from the param map if it has been explicitly set.
-
copy
(extra: Optional[ParamMap] = None) → JP¶ Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.
- Parameters
- extradict, optional
Extra parameters to copy to the new instance
- Returns
JavaParams
Copy of this instance
-
describeTopics
(maxTermsPerTopic: int = 10) → pyspark.sql.dataframe.DataFrame¶ Return the topics described by their top-weighted terms.
New in version 2.0.0.
-
estimatedDocConcentration
() → pyspark.ml.linalg.Vector¶ Value for
LDA.docConcentration
estimated from data. If Online LDA was used andLDA.optimizeDocConcentration
was set to false, then this returns the fixed (given) value for theLDA.docConcentration
parameter.New in version 2.0.0.
-
explainParam
(param: Union[str, pyspark.ml.param.Param]) → str¶ Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
-
explainParams
() → str¶ Returns the documentation of all params with their optionally default values and user-supplied values.
-
extractParamMap
(extra: Optional[ParamMap] = None) → ParamMap¶ Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
- Parameters
- extradict, optional
extra param values
- Returns
- dict
merged param map
-
getCheckpointFiles
() → List[str][source]¶ If using checkpointing and
LDA.keepLastCheckpoint
is set to true, then there may be saved checkpoint files. This method is provided so that users can manage those files.New in version 2.0.0.
- Returns
- list
List of checkpoint files from training
Notes
Removing the checkpoints can cause failures if a partition is lost and is needed by certain
DistributedLDAModel
methods. Reference counting will clean up the checkpoints when this model and derivative data go out of scope.
-
getCheckpointInterval
() → int¶ Gets the value of checkpointInterval or its default value.
-
getDocConcentration
() → List[float]¶ Gets the value of
docConcentration
or its default value.New in version 2.0.0.
-
getFeaturesCol
() → str¶ Gets the value of featuresCol or its default value.
-
getKeepLastCheckpoint
() → bool¶ Gets the value of
keepLastCheckpoint
or its default value.New in version 2.0.0.
-
getLearningDecay
() → float¶ Gets the value of
learningDecay
or its default value.New in version 2.0.0.
-
getLearningOffset
() → float¶ Gets the value of
learningOffset
or its default value.New in version 2.0.0.
-
getMaxIter
() → int¶ Gets the value of maxIter or its default value.
-
getOptimizeDocConcentration
() → bool¶ Gets the value of
optimizeDocConcentration
or its default value.New in version 2.0.0.
-
getOrDefault
(param: Union[str, pyspark.ml.param.Param[T]]) → Union[Any, T]¶ Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
-
getParam
(paramName: str) → pyspark.ml.param.Param¶ Gets a param by its name.
-
getSeed
() → int¶ Gets the value of seed or its default value.
-
getSubsamplingRate
() → float¶ Gets the value of
subsamplingRate
or its default value.New in version 2.0.0.
-
getTopicConcentration
() → float¶ Gets the value of
topicConcentration
or its default value.New in version 2.0.0.
-
getTopicDistributionCol
() → str¶ Gets the value of
topicDistributionCol
or its default value.New in version 2.0.0.
-
hasDefault
(param: Union[str, pyspark.ml.param.Param[Any]]) → bool¶ Checks whether a param has a default value.
-
hasParam
(paramName: str) → bool¶ Tests whether this instance contains a param with a given (string) name.
-
isDefined
(param: Union[str, pyspark.ml.param.Param[Any]]) → bool¶ Checks whether a param is explicitly set by user or has a default value.
-
isDistributed
() → bool¶ Indicates whether this instance is of type DistributedLDAModel
New in version 2.0.0.
-
isSet
(param: Union[str, pyspark.ml.param.Param[Any]]) → bool¶ Checks whether a param is explicitly set by user.
-
classmethod
load
(path: str) → RL¶ Reads an ML instance from the input path, a shortcut of read().load(path).
-
logLikelihood
(dataset: pyspark.sql.dataframe.DataFrame) → float¶ Calculates a lower bound on the log likelihood of the entire corpus. See Equation (16) in the Online LDA paper (Hoffman et al., 2010).
Warning
If this model is an instance of
DistributedLDAModel
(produced whenoptimizer
is set to “em”), this involves collecting a largetopicsMatrix()
to the driver. This implementation may be changed in the future.New in version 2.0.0.
-
logPerplexity
(dataset: pyspark.sql.dataframe.DataFrame) → float¶ Calculate an upper bound on perplexity. (Lower is better.) See Equation (16) in the Online LDA paper (Hoffman et al., 2010).
Warning
If this model is an instance of
DistributedLDAModel
(produced whenoptimizer
is set to “em”), this involves collecting a largetopicsMatrix()
to the driver. This implementation may be changed in the future.New in version 2.0.0.
-
logPrior
() → float[source]¶ Log probability of the current parameter estimate: log P(topics, topic distributions for docs | alpha, eta)
New in version 2.0.0.
-
classmethod
read
() → pyspark.ml.util.JavaMLReader[RL]¶ Returns an MLReader instance for this class.
-
save
(path: str) → None¶ Save this ML instance to the given path, a shortcut of ‘write().save(path)’.
-
set
(param: pyspark.ml.param.Param, value: Any) → None¶ Sets a parameter in the embedded param map.
-
setFeaturesCol
(value: str) → M¶ Sets the value of
featuresCol
.New in version 3.0.0.
-
setTopicDistributionCol
(value: str) → M¶ Sets the value of
topicDistributionCol
.New in version 3.0.0.
-
toLocal
() → pyspark.ml.clustering.LocalLDAModel[source]¶ Convert this distributed model to a local representation. This discards info about the training dataset.
Warning
This involves collecting a large
topicsMatrix()
to the driver.New in version 2.0.0.
-
topicsMatrix
() → pyspark.ml.linalg.Matrix¶ Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.
Warning
If this model is actually a
DistributedLDAModel
instance produced by the Expectation-Maximization (“em”) optimizer, then this method could involve collecting a large amount of data to the driver (on the order of vocabSize x k).New in version 2.0.0.
-
trainingLogLikelihood
() → float[source]¶ Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, Dirichlet hyperparameters)
Notes
This excludes the prior; for that, use
logPrior()
.- Even with
logPrior()
, this is NOT the same as the data log likelihood given the hyperparameters.
- Even with
- This is computed from the topic distributions computed during training. If you call
logLikelihood()
on the same training dataset, the topic distributions will be computed again, possibly giving different results.
New in version 2.0.0.
-
transform
(dataset: pyspark.sql.dataframe.DataFrame, params: Optional[ParamMap] = None) → pyspark.sql.dataframe.DataFrame¶ Transforms the input dataset with optional parameters.
New in version 1.3.0.
- Parameters
- dataset
pyspark.sql.DataFrame
input dataset
- paramsdict, optional
an optional param map that overrides embedded params.
- dataset
- Returns
pyspark.sql.DataFrame
transformed dataset
-
vocabSize
() → int¶ Vocabulary size (number of terms or words in the vocabulary)
New in version 2.0.0.
-
write
() → pyspark.ml.util.JavaMLWriter¶ Returns an MLWriter instance for this ML instance.
Attributes Documentation
-
checkpointInterval
= Param(parent='undefined', name='checkpointInterval', doc='set checkpoint interval (>= 1) or disable checkpoint (-1). E.g. 10 means that the cache will get checkpointed every 10 iterations. Note: this setting will be ignored if the checkpoint directory is not set in the SparkContext.')¶
-
docConcentration
: pyspark.ml.param.Param[List[float]] = Param(parent='undefined', name='docConcentration', doc='Concentration parameter (commonly named "alpha") for the prior placed on documents\' distributions over topics ("theta").')¶
-
featuresCol
= Param(parent='undefined', name='featuresCol', doc='features column name.')¶
-
k
: pyspark.ml.param.Param[int] = Param(parent='undefined', name='k', doc='The number of topics (clusters) to infer. Must be > 1.')¶
-
keepLastCheckpoint
: pyspark.ml.param.Param[bool] = Param(parent='undefined', name='keepLastCheckpoint', doc='(For EM optimizer) If using checkpointing, this indicates whether to keep the last checkpoint. If false, then the checkpoint will be deleted. Deleting the checkpoint can cause failures if a data partition is lost, so set this bit with care.')¶
-
learningDecay
: pyspark.ml.param.Param[float] = Param(parent='undefined', name='learningDecay', doc='Learning rate, set as anexponential decay rate. This should be between (0.5, 1.0] to guarantee asymptotic convergence.')¶
-
learningOffset
: pyspark.ml.param.Param[float] = Param(parent='undefined', name='learningOffset', doc='A (positive) learning parameter that downweights early iterations. Larger values make early iterations count less')¶
-
maxIter
= Param(parent='undefined', name='maxIter', doc='max number of iterations (>= 0).')¶
-
optimizeDocConcentration
: pyspark.ml.param.Param[bool] = Param(parent='undefined', name='optimizeDocConcentration', doc='Indicates whether the docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training.')¶
-
optimizer
: pyspark.ml.param.Param[str] = Param(parent='undefined', name='optimizer', doc='Optimizer or inference algorithm used to estimate the LDA model. Supported: online, em')¶
-
params
¶ Returns all params ordered by name. The default implementation uses
dir()
to get all attributes of typeParam
.
-
seed
= Param(parent='undefined', name='seed', doc='random seed.')¶
-
subsamplingRate
: pyspark.ml.param.Param[float] = Param(parent='undefined', name='subsamplingRate', doc='Fraction of the corpus to be sampled and used in each iteration of mini-batch gradient descent, in range (0, 1].')¶
-
topicConcentration
: pyspark.ml.param.Param[float] = Param(parent='undefined', name='topicConcentration', doc='Concentration parameter (commonly named "beta" or "eta") for the prior placed on topic\' distributions over terms.')¶
-
topicDistributionCol
: pyspark.ml.param.Param[str] = Param(parent='undefined', name='topicDistributionCol', doc='Output column with estimates of the topic mixture distribution for each document (often called "theta" in the literature). Returns a vector of zeros for an empty document.')¶
-