optuna.integration

class optuna.integration.ChainerPruningExtension(trial, observation_key, pruner_trigger)[source]

Chainer extension to prune unpromising trials.

See the example if you want to add a pruning extension which observes validation accuracy of a Chainer Trainer.

Parameters

  • trial – A Trial corresponding to the current evaluation of the objective function.

  • observation_key – An evaluation metric for pruning, e.g., main/loss and validation/main/accuracy. Please refer to chainer.Reporter reference for further details.

  • pruner_trigger –

    A trigger to execute pruning. pruner_trigger is an instance of IntervalTrigger or ManualScheduleTrigger. IntervalTrigger can be specified by a tuple of the interval length and its unit like (1, 'epoch').

class optuna.integration.ChainerMNStudy(study, comm)[source]

A wrapper of Study to incorporate Optuna with ChainerMN.

See also

ChainerMNStudy provides the same interface as Study. Please refer to optuna.study.Study for further details.

See the example if you want to optimize an objective function that trains neural network written with ChainerMN.

Parameters
optimize(func, n_trials=None, timeout=None, catch=())[source]

Optimize an objective function.

This method provides the same interface as optuna.study.Study.optimize() except the absence of n_jobs argument.

class optuna.integration.PyCmaSampler(x0=None, sigma0=None, cma_stds=None, seed=None, cma_opts=None, n_startup_trials=1, independent_sampler=None, warn_independent_sampling=True)[source]

A Sampler using cma library as the backend.

Example

Optimize a simple quadratic function by using PyCmaSampler.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -1, 1)
    y = trial.suggest_int('y', -1, 1)
    return x**2 + y

sampler = optuna.integration.PyCmaSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=20)

Note that parallel execution of trials may affect the optimization performance of CMA-ES, especially if the number of trials running in parallel exceeds the population size.

Note

CmaEsSampler is deprecated and renamed to PyCmaSampler in v2.0.0. Please use PyCmaSampler instead of CmaEsSampler.

Parameters

  • x0 – A dictionary of an initial parameter values for CMA-ES. By default, the mean of low and high for each distribution is used. Please refer to cma.CMAEvolutionStrategy for further details of x0.

  • sigma0 – Initial standard deviation of CMA-ES. By default, sigma0 is set to min_range / 6, where min_range denotes the minimum range of the distributions in the search space. If distribution is categorical, min_range is len(choices) - 1. Please refer to cma.CMAEvolutionStrategy for further details of sigma0.

  • cma_stds – A dictionary of multipliers of sigma0 for each parameters. The default value is 1.0. Please refer to cma.CMAEvolutionStrategy for further details of cma_stds.

  • seed – A random seed for CMA-ES.

  • cma_opts –

    Options passed to the constructor of cma.CMAEvolutionStrategy class.

    Note that BoundaryHandler, bounds, CMA_stds and seed arguments in cma_opts will be ignored because it is added by PyCmaSampler automatically.

  • n_startup_trials – The independent sampling is used instead of the CMA-ES algorithm until the given number of trials finish in the same study.

  • independent_sampler –

    A BaseSampler instance that is used for independent sampling. The parameters not contained in the relative search space are sampled by this sampler. The search space for PyCmaSampler is determined by intersection_search_space().

    If None is specified, RandomSampler is used as the default.

    See also

    optuna.samplers module provides built-in independent samplers such as RandomSampler and TPESampler.

  • warn_independent_sampling –

    If this is True, a warning message is emitted when the value of a parameter is sampled by using an independent sampler.

    Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case.

reseed_rng() → None[source]

Reseed sampler’s random number generator.

This method is called by the Study instance if trials are executed in parallel with the option n_jobs>1. In that case, the sampler instance will be replicated including the state of the random number generator, and they may suggest the same values. To prevent this issue, this method assigns a different seed to each random number generator.

class optuna.integration.CmaEsSampler(x0=None, sigma0=None, cma_stds=None, seed=None, cma_opts=None, n_startup_trials=1, independent_sampler=None, warn_independent_sampling=True)[source]

Wrapper class of PyCmaSampler for backward compatibility.

Deprecated since version 2.0.0: This class is renamed to PyCmaSampler.

class optuna.integration.FastAIPruningCallback(learn, trial, monitor)[source]

FastAI callback to prune unpromising trials for fastai.

Note

This callback is for fastai<2.0, not the coming version developed in fastai/fastai_dev.

See the example if you want to add a pruning callback which monitors validation loss of a Learner.

Example

Register a pruning callback to learn.fit and learn.fit_one_cycle.

learn.fit(n_epochs, callbacks=[FastAIPruningCallback(learn, trial, 'valid_loss')])
learn.fit_one_cycle(
    n_epochs, cyc_len, max_lr,
    callbacks=[FastAIPruningCallback(learn, trial, 'valid_loss')])
Parameters
class optuna.integration.PyTorchIgnitePruningHandler(trial, metric, trainer)[source]

PyTorch Ignite handler to prune unpromising trials.

See the example if you want to add a pruning handler which observes validation accuracy.

Parameters

  • trial – A Trial corresponding to the current evaluation of the objective function.

  • metric – A name of metric for pruning, e.g., accuracy and loss.

  • trainer – A trainer engine of PyTorch Ignite. Please refer to ignite.engine.Engine reference for further details.

class optuna.integration.KerasPruningCallback(trial, monitor, interval=1)[source]

Keras callback to prune unpromising trials.

See the example if you want to add a pruning callback which observes validation accuracy.

Parameters

  • trial – A Trial corresponding to the current evaluation of the objective function.

  • monitor – An evaluation metric for pruning, e.g., val_loss and val_accuracy. Please refer to keras.Callback reference for further details.

  • interval – Check if trial should be pruned every n-th epoch. By default interval=1 and pruning is performed after every epoch. Increase interval to run several epochs faster before applying pruning.

class optuna.integration.LightGBMPruningCallback(trial, metric, valid_name='valid_0')[source]

Callback for LightGBM to prune unpromising trials.

See the example if you want to add a pruning callback which observes AUC of a LightGBM model.

Parameters

  • trial – A Trial corresponding to the current evaluation of the objective function.

  • metric – An evaluation metric for pruning, e.g., binary_error and multi_error. Please refer to LightGBM reference for further details.

  • valid_name – The name of the target validation. Validation names are specified by valid_names option of train method. If omitted, valid_0 is used which is the default name of the first validation. Note that this argument will be ignored if you are calling cv method instead of train method.

optuna.integration.lightgbm.train(*args: Any, **kwargs: Any) → Any[source]

Wrapper of LightGBM Training API to tune hyperparameters.

It tunes important hyperparameters (e.g., min_child_samples and feature_fraction) in a stepwise manner. It is a drop-in replacement for lightgbm.train(). See a simple example of LightGBM Tuner which optimizes the validation log loss of cancer detection.

train() is a wrapper function of LightGBMTuner. To use feature in Optuna such as suspended/resumed optimization and/or parallelization, refer to LightGBMTuner instead of this function.

Arguments and keyword arguments for lightgbm.train() can be passed.

Note

Added in v0.18.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v0.18.0.

class optuna.integration.lightgbm.LightGBMTuner(params: Dict[str, Any], train_set: lgb.Dataset, num_boost_round: int = 1000, valid_sets: Optional[VALID_SET_TYPE] = None, valid_names: Optional[Any] = None, fobj: Optional[Callable[[…], Any]] = None, feval: Optional[Callable[[…], Any]] = None, feature_name: str = 'auto', categorical_feature: str = 'auto', early_stopping_rounds: Optional[int] = None, evals_result: Optional[Dict[Any, Any]] = None, verbose_eval: Union[bool, int, None] = True, learning_rates: Optional[List[float]] = None, keep_training_booster: Optional[bool] = False, callbacks: Optional[List[Callable[[…], Any]]] = None, time_budget: Optional[int] = None, sample_size: Optional[int] = None, study: Optional[optuna.study.Study] = None, optuna_callbacks: Optional[List[Callable[[optuna.study.Study, optuna.trial._frozen.FrozenTrial], None]]] = None, model_dir: Optional[str] = None, verbosity: Optional[int] = 1)[source]

Hyperparameter tuner for LightGBM.

It optimizes the following hyperparameters in a stepwise manner: lambda_l1, lambda_l2, num_leaves, feature_fraction, bagging_fraction, bagging_freq and min_child_samples.

You can find the details of the algorithm and benchmark results in this blog article by Kohei Ozaki, a Kaggle Grandmaster.

Arguments and keyword arguments for lightgbm.train() can be passed. The arguments that only LightGBMTuner has are listed below:

Parameters

  • time_budget – A time budget for parameter tuning in seconds.

  • study – A Study instance to store optimization results. The Trial instances in it has the following user attributes: elapsed_secs is the elapsed time since the optimization starts. average_iteration_time is the average time of iteration to train the booster model in the trial. lgbm_params is a JSON-serialized dictionary of LightGBM parameters used in the trial.

  • optuna_callbacks – List of Optuna callback functions that are invoked at the end of each trial. Each function must accept two parameters with the following types in this order: Study and FrozenTrial. Please note that this is not a callbacks argument of lightgbm.train() .

  • model_dir – A directory to save boosters. By default, it is set to None and no boosters are saved. Please set shared directory (e.g., directories on NFS) if you want to access get_best_booster() in distributed environments. Otherwise, it may raise ValueError. If the directory does not exist, it will be created. The filenames of the boosters will be {model_dir}/{trial_number}.pkl (e.g., ./boosters/0.pkl).

Note

Added in v1.5.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.5.0.

property best_booster

Return the best booster.

Deprecated since version 1.4.0: Please get the best booster via get_best_booster instead.

property best_params

Return parameters of the best booster.

property best_score

Return the score of the best booster.

get_best_booster() → lgb.Booster[source]

Return the best booster.

If the best booster cannot be found, ValueError will be raised. To prevent the errors, please save boosters by specifying the model_dir arguments of __init__() when you resume tuning or you run tuning in parallel.

run() → None

Perform the hyperparameter-tuning with given parameters.

class optuna.integration.lightgbm.LightGBMTunerCV(params: Dict[str, Any], train_set: lgb.Dataset, num_boost_round: int = 1000, folds: Union[Generator[Tuple[int, int], None, None], Iterator[Tuple[int, int]], BaseCrossValidator, None] = None, nfold: int = 5, stratified: bool = True, shuffle: bool = True, fobj: Optional[Callable[[…], Any]] = None, feval: Optional[Callable[[…], Any]] = None, feature_name: str = 'auto', categorical_feature: str = 'auto', early_stopping_rounds: Optional[int] = None, fpreproc: Optional[Callable[[…], Any]] = None, verbose_eval: Union[bool, int, None] = True, show_stdv: bool = True, seed: int = 0, callbacks: Optional[List[Callable[[…], Any]]] = None, time_budget: Optional[int] = None, sample_size: Optional[int] = None, study: Optional[optuna.study.Study] = None, optuna_callbacks: Optional[List[Callable[[optuna.study.Study, optuna.trial._frozen.FrozenTrial], None]]] = None, verbosity: int = 1)[source]

Hyperparameter tuner for LightGBM with cross-validation.

It employs the same stepwise approach as LightGBMTuner. LightGBMTunerCV invokes lightgbm.cv() to train and validate boosters while LightGBMTuner invokes lightgbm.train(). See a simple example which optimizes the validation log loss of cancer detection.

Arguments and keyword arguments for lightgbm.cv() can be passed except metrics, init_model and eval_train_metric. The arguments that only LightGBMTunerCV has are listed below:

Parameters

  • time_budget – A time budget for parameter tuning in seconds.

  • study – A Study instance to store optimization results. The Trial instances in it has the following user attributes: elapsed_secs is the elapsed time since the optimization starts. average_iteration_time is the average time of iteration to train the booster model in the trial. lgbm_params is a JSON-serialized dictionary of LightGBM parameters used in the trial.

  • optuna_callbacks – List of Optuna callback functions that are invoked at the end of each trial. Each function must accept two parameters with the following types in this order: Study and FrozenTrial. Please note that this is not a callbacks argument of lightgbm.train() .

Note

Added in v1.5.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.5.0.

property best_params

Return parameters of the best booster.

property best_score

Return the score of the best booster.

run() → None

Perform the hyperparameter-tuning with given parameters.

class optuna.integration.MLflowCallback(tracking_uri=None, metric_name='value')[source]

Callback to track Optuna trials with MLflow.

This callback adds relevant information that is tracked by Optuna to MLflow. The MLflow experiment will be named after the Optuna study name.

Example

Add MLflow callback to Optuna optimization.

import optuna
from optuna.integration.mlflow import MLflowCallback

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

mlflc = MLflowCallback(
    tracking_uri=YOUR_TRACKING_URI,
    metric_name='my metric score',
)

study = optuna.create_study(study_name='my_study')
study.optimize(objective, n_trials=10, callbacks=[mlflc])
Parameters

  • tracking_uri –

    The URI of the MLflow tracking server.

    Please refer to mlflow.set_tracking_uri for more details.

  • metric_name – Name of the metric. Since the metric itself is just a number, metric_name can be used to give it a name. So you know later if it was roc-auc or accuracy.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

class optuna.integration.MXNetPruningCallback(trial, eval_metric)[source]

MXNet callback to prune unpromising trials.

See the example if you want to add a pruning callback which observes accuracy.

Parameters

  • trial – A Trial corresponding to the current evaluation of the objective function.

  • eval_metric – An evaluation metric name for pruning, e.g., cross-entropy and accuracy. If using default metrics like mxnet.metrics.Accuracy, use it’s default metric name. For custom metrics, use the metric_name provided to constructor. Please refer to mxnet.metrics reference for further details.

class optuna.integration.PyTorchLightningPruningCallback(trial: optuna.trial._trial.Trial, monitor: str)[source]

PyTorch Lightning callback to prune unpromising trials.

See the example if you want to add a pruning callback which observes accuracy.

Parameters

  • trial – A Trial corresponding to the current evaluation of the objective function.

  • monitor – An evaluation metric for pruning, e.g., val_loss or val_acc. The metrics are obtained from the returned dictionaries from e.g. pytorch_lightning.LightningModule.training_step or pytorch_lightning.LightningModule.validation_end and the names thus depend on how this dictionary is formatted.

class optuna.integration.SkoptSampler(independent_sampler=None, warn_independent_sampling=True, skopt_kwargs=None, n_startup_trials=1)[source]

Sampler using Scikit-Optimize as the backend.

Example

Optimize a simple quadratic function by using SkoptSampler.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    y = trial.suggest_int('y', 0, 10)
    return x**2 + y

sampler = optuna.integration.SkoptSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=10)
Parameters

  • independent_sampler –

    A BaseSampler instance that is used for independent sampling. The parameters not contained in the relative search space are sampled by this sampler. The search space for SkoptSampler is determined by intersection_search_space().

    If None is specified, RandomSampler is used as the default.

    See also

    optuna.samplers module provides built-in independent samplers such as RandomSampler and TPESampler.

  • warn_independent_sampling –

    If this is True, a warning message is emitted when the value of a parameter is sampled by using an independent sampler.

    Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case.

  • skopt_kwargs –

    Keyword arguments passed to the constructor of skopt.Optimizer class.

    Note that dimensions argument in skopt_kwargs will be ignored because it is added by SkoptSampler automatically.

  • n_startup_trials – The independent sampling is used until the given number of trials finish in the same study.

reseed_rng() → None[source]

Reseed sampler’s random number generator.

This method is called by the Study instance if trials are executed in parallel with the option n_jobs>1. In that case, the sampler instance will be replicated including the state of the random number generator, and they may suggest the same values. To prevent this issue, this method assigns a different seed to each random number generator.

class optuna.integration.TensorFlowPruningHook(trial, estimator, metric, run_every_steps)[source]

TensorFlow SessionRunHook to prune unpromising trials.

See the example if you want to add a pruning hook to TensorFlow’s estimator.

Parameters

  • trial – A Trial corresponding to the current evaluation of the objective function.

  • estimator – An estimator which you will use.

  • metric – An evaluation metric for pruning, e.g., accuracy and loss.

  • run_every_steps – An interval to watch the summary file.

class optuna.integration.TFKerasPruningCallback(trial, monitor)[source]

tf.keras callback to prune unpromising trials.

This callback is intend to be compatible for TensorFlow v1 and v2, but only tested with TensorFlow v1.

See the example if you want to add a pruning callback which observes the validation accuracy.

Parameters

  • trial – A Trial corresponding to the current evaluation of the objective function.

  • monitor – An evaluation metric for pruning, e.g., val_loss or val_acc.

class optuna.integration.XGBoostPruningCallback(trial, observation_key)[source]

Callback for XGBoost to prune unpromising trials.

See the example if you want to add a pruning callback which observes validation AUC of a XGBoost model.

Parameters

  • trial – A Trial corresponding to the current evaluation of the objective function.

  • observation_key – An evaluation metric for pruning, e.g., validation-error and validation-merror. When using the Scikit-Learn API, the index number of eval_set must be included in the observation_key, e.g., validation_0-error and validation_0-merror. Please refer to eval_metric in XGBoost reference for further details.

class optuna.integration.OptunaSearchCV(estimator, param_distributions, cv=5, enable_pruning=False, error_score=nan, max_iter=1000, n_jobs=1, n_trials=10, random_state=None, refit=True, return_train_score=False, scoring=None, study=None, subsample=1.0, timeout=None, verbose=0)[source]

Hyperparameter search with cross-validation.

Warning

This feature is experimental. The interface may be changed in the future.

Parameters

  • estimator – Object to use to fit the data. This is assumed to implement the scikit-learn estimator interface. Either this needs to provide score, or scoring must be passed.

  • param_distributions – Dictionary where keys are parameters and values are distributions. Distributions are assumed to implement the optuna distribution interface.

  • cv –

    Cross-validation strategy. Possible inputs for cv are:

    • integer to specify the number of folds in a CV splitter,

    • a CV splitter,

    • an iterable yielding (train, validation) splits as arrays of indices.

    For integer, if estimator is a classifier and y is either binary or multiclass, sklearn.model_selection.StratifiedKFold is used. otherwise, sklearn.model_selection.KFold is used.

  • enable_pruning – If True, pruning is performed in the case where the underlying estimator supports partial_fit.

  • error_score – Value to assign to the score if an error occurs in fitting. If ‘raise’, the error is raised. If numeric, sklearn.exceptions.FitFailedWarning is raised. This does not affect the refit step, which will always raise the error.

  • max_iter – Maximum number of epochs. This is only used if the underlying estimator supports partial_fit.

  • n_jobs – Number of parallel jobs. -1 means using all processors.

  • n_trials – Number of trials. If None, there is no limitation on the number of trials. If timeout is also set to None, the study continues to create trials until it receives a termination signal such as Ctrl+C or SIGTERM. This trades off runtime vs quality of the solution.

  • random_state – Seed of the pseudo random number generator. If int, this is the seed used by the random number generator. If numpy.random.RandomState object, this is the random number generator. If None, the global random state from numpy.random is used.

  • refit – If True, refit the estimator with the best found hyperparameters. The refitted estimator is made available at the best_estimator_ attribute and permits using predict directly.

  • return_train_score – If True, training scores will be included. Computing training scores is used to get insights on how different hyperparameter settings impact the overfitting/underfitting trade-off. However computing training scores can be computationally expensive and is not strictly required to select the hyperparameters that yield the best generalization performance.

  • scoring – String or callable to evaluate the predictions on the validation data. If None, score on the estimator is used.

  • study – Study corresponds to the optimization task. If None, a new study is created.

  • subsample –

    Proportion of samples that are used during hyperparameter search.

    • If int, then draw subsample samples.

    • If float, then draw subsample * X.shape[0] samples.

  • timeout – Time limit in seconds for the search of appropriate models. If None, the study is executed without time limitation. If n_trials is also set to None, the study continues to create trials until it receives a termination signal such as Ctrl+C or SIGTERM. This trades off runtime vs quality of the solution.

  • verbose – Verbosity level. The higher, the more messages.

best_estimator\_

Estimator that was chosen by the search. This is present only if refit is set to True.

n_splits\_

Number of cross-validation splits.

refit_time\_

Time for refitting the best estimator. This is present only if refit is set to True.

sample_indices\_

Indices of samples that are used during hyperparameter search.

scorer\_

Scorer function.

study\_

Actual study.

Examples

import optuna
from sklearn.datasets import load_iris
from sklearn.svm import SVC

clf = SVC(gamma='auto')
param_distributions = {
    'C': optuna.distributions.LogUniformDistribution(1e-10, 1e+10)
}
optuna_search = optuna.integration.OptunaSearchCV(
    clf,
    param_distributions
)
X, y = load_iris(return_X_y=True)
optuna_search.fit(X, y)
y_pred = optuna_search.predict(X)
property best_index_

Index which corresponds to the best candidate parameter setting.

property best_params_

Parameters of the best trial in the Study.

property best_score_

Mean cross-validated score of the best estimator.

property best_trial_

Best trial in the Study.

property classes_

Class labels.

property decision_function

Call decision_function on the best estimator.

This is available only if the underlying estimator supports decision_function and refit is set to True.

fit(X, y=None, groups=None, **fit_params)[source]

Run fit with all sets of parameters.

Parameters

  • X – Training data.

  • y – Target variable.

  • groups – Group labels for the samples used while splitting the dataset into train/validation set.

  • **fit_params – Parameters passed to fit on the estimator.

Returns

Return self.

Return type

self

property inverse_transform

Call inverse_transform on the best estimator.

This is available only if the underlying estimator supports inverse_transform and refit is set to True.

property n_trials_

Actual number of trials.

property predict

Call predict on the best estimator.

This is available only if the underlying estimator supports predict and refit is set to True.

property predict_log_proba

Call predict_log_proba on the best estimator.

This is available only if the underlying estimator supports predict_log_proba and refit is set to True.

property predict_proba

Call predict_proba on the best estimator.

This is available only if the underlying estimator supports predict_proba and refit is set to True.

score(X, y=None)[source]

Return the score on the given data.

Parameters

  • X – Data.

  • y – Target variable.

Returns

Scaler score.

Return type

score

property score_samples

Call score_samples on the best estimator.

This is available only if the underlying estimator supports score_samples and refit is set to True.

property set_user_attr

Call set_user_attr on the Study.

property transform

Call transform on the best estimator.

This is available only if the underlying estimator supports transform and refit is set to True.

property trials_

All trials in the Study.

property trials_dataframe

Call trials_dataframe on the Study.

property user_attrs_

User attributes in the Study.

class optuna.integration.AllenNLPExecutor(trial: optuna.trial._trial.Trial, config_file: str, serialization_dir: str, metrics: str = 'best_validation_accuracy', *, include_package: Union[str, List[str], None] = None)[source]

AllenNLP extension to use optuna with Jsonnet config file.

This feature is experimental since AllenNLP major release will come soon. The interface may change without prior notice to correspond to the update.

See the examples of objective function and config file.

Parameters

  • trial – A Trial corresponding to the current evaluation of the objective function.

  • config_file – Config file for AllenNLP. Hyperparameters should be masked with std.extVar. Please refer to the config example.

  • serialization_dir – A path which model weights and logs are saved.

  • metrics – An evaluation metric for the result of objective.

  • include_package – Additional packages to include. For more information, please see AllenNLP documentation.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

run() → float[source]

Train a model using AllenNLP.

optuna.integration.allennlp.dump_best_config(input_config_file: str, output_config_file: str, study: optuna.study.Study) → None[source]

Save JSON config file after updating with parameters from the best trial in the study.

Parameters

  • input_config_file – Input Jsonnet config file used with AllenNLPExecutor.

  • output_config_file – Output JSON config file.

  • study – Instance of Study. Note that optimize() must have been called.