Optuna: A hyperparameter optimization framework¶

Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, define-by-run style user API. Thanks to our define-by-run API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters.

Key Features¶

Optuna has modern functionalities as follows:

Basic Concepts¶

We use the terms study and trial as follows:

Study: optimization based on an objective function
Trial: a single execution of the objective function

Please refer to sample code below. The goal of a study is to find out the optimal set of hyperparameter values (e.g., classifier and svm_c) through multiple trials (e.g., n_trials=100). Optuna is a framework designed for the automation and the acceleration of the optimization studies.

import ...

# Define an objective function to be minimized.
def objective(trial):

    # Invoke suggest methods of a Trial object to generate hyperparameters.
    regressor_name = trial.suggest_categorical('classifier', ['SVR', 'RandomForest'])
    if regressor_name == 'SVR':
        svr_c = trial.suggest_loguniform('svr_c', 1e-10, 1e10)
        regressor_obj = sklearn.svm.SVR(C=svr_c)
    else:
        rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32)
        regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth)

    X, y = sklearn.datasets.load_boston(return_X_y=True)
    X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0)

    regressor_obj.fit(X_train, y_train)
    y_pred = regressor_obj.predict(X_val)

    error = sklearn.metrics.mean_squared_error(y_val, y_pred)

    return error  # An objective value linked with the Trial object.

study = optuna.create_study()  # Create a new study.
study.optimize(objective, n_trials=100)  # Invoke optimization of the objective function.

Communication¶

GitHub Issues for bug reports, feature requests and questions.
Gitter for interactive chat with developers.
Stack Overflow for questions.

Contribution¶

Any contributions to Optuna are welcome! When you send a pull request, please follow the contribution guide.

License¶

MIT License (see LICENSE).

Reference¶

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In KDD (arXiv).

Installation¶

Optuna supports Python 3.5 or newer.

We recommend to install Optuna via pip:

$ pip install optuna

You can also install the development version of Optuna from master branch of Git repository:

$ pip install git+https://github.com/optuna/optuna.git

You can also install Optuna via conda:

$ conda install -c conda-forge optuna

Tutorial¶

First Optimization¶

Quadratic Function Example¶

Usually, Optuna is used to optimize hyper-parameters, but as an example, let us directly optimize a quadratic function in an IPython shell.

import optuna

The objective function is what will be optimized.

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

This function returns the value of $(x - 2)^2$. Our goal is to find the value of x that minimizes the output of the objective function. This is the “optimization.” During the optimization, Optuna repeatedly calls and evaluates the objective function with different values of x.

A Trial object corresponds to a single execution of the objective function and is internally instantiated upon each invocation of the function.

The suggest APIs (for example, suggest_uniform()) are called inside the objective function to obtain parameters for a trial. suggest_uniform() selects parameters uniformly within the range provided. In our example, from -10 to 10.

To start the optimization, we create a study object and pass the objective function to method optimize() as follows.

study = optuna.create_study()
study.optimize(objective, n_trials=100)

Out:

[I 2020-04-08 10:42:09,028] Trial 0 finished with value: 25.77382032395108 with parameters: {'x': 7.076792326257898}. Best is trial 0 with value: 25.77382032395108.
[I 2020-04-08 10:42:09,064] Trial 1 finished with value: 1.5189812248635903 with parameters: {'x': 0.7675304365366298}. Best is trial 1 with value: 1.5189812248635903.
[I 2020-04-08 10:42:09,106] Trial 2 finished with value: 34.4074691838153 with parameters: {'x': -3.865788027521562}. Best is trial 1 with value: 1.5189812248635903.
[I 2020-04-08 10:42:09,145] Trial 3 finished with value: 3.3601305753722657 with parameters: {'x': 3.8330658949891205}. Best is trial 1 with value: 1.5189812248635903.
[I 2020-04-08 10:42:09,185] Trial 4 finished with value: 61.16797535698886 with parameters: {'x': -5.820995803412048}. Best is trial 1 with value: 1.5189812248635903.
[I 2020-04-08 10:42:09,228] Trial 5 finished with value: 90.08665552769618 with parameters: {'x': -7.491399028999686}. Best is trial 1 with value: 1.5189812248635903.
[I 2020-04-08 10:42:09,274] Trial 6 finished with value: 25.254236332163032 with parameters: {'x': 7.025359323686519}. Best is trial 1 with value: 1.5189812248635903.
...
[I 2020-04-08 10:42:14,237] Trial 99 finished with value: 0.5227007740782738 with parameters: {'x': 2.7229804797352926}. Best is trial 67 with value: 2.916284393762304e-06.

You can get the best parameter as follows.

study.best_params

Out:

{'x': 2.001707713205946}

We can see that Optuna found the best x value 2.001707713205946, which is close to the optimal value of 2.

Note

When used to search for hyper-parameters in machine learning, usually the objective function would return the loss or accuracy of the model.

Study Object¶

Let us clarify the terminology in Optuna as follows:

Trial: A single call of the objective function
Study: An optimization session, which is a set of trials
Parameter: A variable whose value is to be optimized, such as x in the above example

In Optuna, we use the study object to manage optimization. Method create_study() returns a study object. A study object has useful properties for analyzing the optimization outcome.

To get the best parameter:

study.best_params

Out:

{'x': 2.001707713205946}

To get the best value:

study.best_value

Out:

2.916284393762304e-06

To get the best trial:

study.best_trial

Out:

FrozenTrial(number=67, value=2.916284393762304e-06, datetime_start=datetime.datetime(2020, 4, 8, 10, 42, 12, 595884), datetime_complete=datetime.datetime(2020, 4, 8, 10, 42, 12, 639969), params={'x': 2.001707713205946}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=67, state=TrialState.COMPLETE)

To get all trials:

study.trials

Out:

[FrozenTrial(number=0, value=25.77382032395108, datetime_start=datetime.datetime(2020, 4, 8, 10, 42, 8, 987277), datetime_complete=datetime.datetime(2020, 4, 8, 10, 42, 9, 27959), params={'x': 7.076792326257898}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=0, state=TrialState.COMPLETE),
 ...
 user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=99, state=TrialState.COMPLETE)]

To get the number of trials:

len(study.trials)

Out:

By executing optimize() again, we can continue the optimization.

study.optimize(objective, n_trials=100)

To get the updated number of trials:

len(study.trials)

Out:

Advanced Configurations¶

Defining Parameter Spaces¶

Optuna supports five kinds of parameters.

def objective(trial):
    # Categorical parameter
    optimizer = trial.suggest_categorical('optimizer', ['MomentumSGD', 'Adam'])

    # Int parameter
    num_layers = trial.suggest_int('num_layers', 1, 3)

    # Uniform parameter
    dropout_rate = trial.suggest_uniform('dropout_rate', 0.0, 1.0)

    # Loguniform parameter
    learning_rate = trial.suggest_loguniform('learning_rate', 1e-5, 1e-2)

    # Discrete-uniform parameter
    drop_path_rate = trial.suggest_discrete_uniform('drop_path_rate', 0.0, 1.0, 0.1)

    ...

Branches and Loops¶

You can use branches or loops depending on the parameter values.

def objective(trial):
    classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
    if classifier_name == 'SVC':
        svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
        classifier_obj = sklearn.svm.SVC(C=svc_c)
    else:
        rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
        classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth)

    ...

def create_model(trial):
    n_layers = trial.suggest_int('n_layers', 1, 3)

    layers = []
    for i in range(n_layers):
        n_units = int(trial.suggest_loguniform('n_units_l{}'.format(i), 4, 128))
        layers.append(L.Linear(None, n_units))
        layers.append(F.relu)
    layers.append(L.Linear(None, 10))

    return chainer.Sequential(*layers)

Please also refer to examples.

Note on the Number of Parameters¶

The difficulty of optimization increases roughly exponentially with regard to the number of parameters. That is, the number of necessary trials increases exponentially when you increase the number of parameters, so it is recommended to not add unimportant parameters.

Arguments for Study.optimize¶

The method optimize() (and optuna study optimize CLI command as well) has several useful options such as timeout. For details, please refer to the API reference for optimize().

FYI: If you give neither n_trials nor timeout options, the optimization continues until it receives a termination signal such as Ctrl+C or SIGTERM. This is useful for use cases such as when it is hard to estimate the computational costs required to optimize your objective function.

Saving/Resuming Study with RDB Backend¶

An RDB backend enables persistent experiments (i.e., to save and resume a study) as well as access to history of studies. In addition, we can run multi-node optimization tasks with this feature, which is described in Distributed Optimization.

In this section, let’s try simple examples running on a local environment with SQLite DB.

Note

You can also utilize other RDB backends, e.g., PostgreSQL or MySQL, by setting the storage argument to the DB’s URL. Please refer to SQLAlchemy’s document for how to set up the URL.

New Study¶

We can create a persistent study by calling create_study() function as follows. An SQLite file example.db is automatically initialized with a new study record.

import optuna
study_name = 'example-study'  # Unique identifier of the study.
study = optuna.create_study(study_name=study_name, storage='sqlite:///example.db')

To run a study, call optimize() method passing an objective function.

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

study.optimize(objective, n_trials=3)

Resume Study¶

To resume a study, instantiate a Study object passing the study name example-study and the DB URL sqlite:///example.db.

study = optuna.create_study(study_name='example-study', storage='sqlite:///example.db', load_if_exists=True)
study.optimize(objective, n_trials=3)

Experimental History¶

We can access histories of studies and trials via the Study class. For example, we can get all trials of example-study as:

import optuna
study = optuna.create_study(study_name='example-study', storage='sqlite:///example.db', load_if_exists=True)
df = study.trials_dataframe(attrs=('number', 'value', 'params', 'state'))

The method trials_dataframe() returns a pandas dataframe like:

print(df)

Out:

   number       value  params_x     state
     0   25.301959 -3.030105  COMPLETE
     1    1.406223  0.814157  COMPLETE
     2   44.010366 -4.634031  COMPLETE
     3   55.872181  9.474770  COMPLETE
     4  113.039223 -8.631991  COMPLETE
     5   57.319570  9.570969  COMPLETE

A Study object also provides properties such as trials, best_value, best_params (see also First Optimization).

study.best_params  # Get best parameters for the objective function.
study.best_value  # Get best objective value.
study.best_trial  # Get best trial's information.
study.trials  # Get all trials' information.

Distributed Optimization¶

There is no complicated setup but just sharing the same study name among nodes/processes.

First, create a shared study using optuna create-study command (or using optuna.create_study() in a Python script).

$ optuna create-study --study-name "distributed-example" --storage "sqlite:///example.db"
[I 2018-10-31 18:21:57,885] A new study created with name: distributed-example

Then, write an optimization script. Let’s assume that foo.py contains the following code.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

if __name__ == '__main__':
    study = optuna.load_study(study_name='distributed-example', storage='sqlite:///example.db')
    study.optimize(objective, n_trials=100)

Finally, run the shared study from multiple processes. For example, run Process 1 in a terminal, and do Process 2 in another one. They get parameter suggestions based on shared trials’ history.

Process 1:

$ python foo.py
[I 2018-10-31 18:46:44,308] Finished a trial resulted in value: 1.1097007755908204. Current best value is 0.00020881104123229936 with parameters: {'x': 2.014450295541348}.
[I 2018-10-31 18:46:44,361] Finished a trial resulted in value: 0.5186699439824186. Current best value is 0.00020881104123229936 with parameters: {'x': 2.014450295541348}.
...

Process 2 (the same command as process 1):

$ python foo.py
[I 2018-10-31 18:47:02,912] Finished a trial resulted in value: 29.821448668796563. Current best value is 0.00020881104123229936 with parameters: {'x': 2.014450295541348}.
[I 2018-10-31 18:47:02,968] Finished a trial resulted in value: 0.7962498978463782. Current best value is 0.00020881104123229936 with parameters: {'x': 2.014450295541348}.
...

Note

We do not recommend SQLite for large scale distributed optimizations because it may cause serious performance issues. Please consider to use another database engine like PostgreSQL or MySQL.

Note

Please avoid putting the SQLite database on NFS when running distributed optimizations. See also: https://www.sqlite.org/faq.html#q5

Command-Line Interface¶

Command	Description
create-study	Create a new study.
delete-study	Delete a specified study.
dashboard	Launch web dashboard (beta).
storage upgrade	Upgrade the schema of a storage.
studies	Show a list of studies.
study optimize	Start optimization of a study.
study set-user-attr	Set a user attribute to a study.

Optuna provides command-line interface as shown in the above table.

Let us assume you are not in IPython shell and writing Python script files instead. It is totally fine to write scripts like the following:

import optuna


def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2


if __name__ == '__main__':
    study = optuna.create_study()
    study.optimize(objective, n_trials=100)
    print('Best value: {} (params: {})\n'.format(study.best_value, study.best_params))

However, we can reduce boilerplate codes by using our optuna command. Let us assume that foo.py contains only the following code.

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

Even so, we can invoke the optimization as follows. (Don’t care about --storage sqlite:///example.db for now, which is described in Saving/Resuming Study with RDB Backend.)

$ cat foo.py
def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

$ STUDY_NAME=`optuna create-study --storage sqlite:///example.db`
$ optuna study optimize foo.py objective --n-trials=100 --storage sqlite:///example.db --study-name $STUDY_NAME
[I 2018-05-09 10:40:25,196] Finished a trial resulted in value: 54.353767789264026. Current best value is 54.353767789264026 with parameters: {'x': -5.372500782588228}.
[I 2018-05-09 10:40:25,197] Finished a trial resulted in value: 15.784266965526376. Current best value is 15.784266965526376 with parameters: {'x': 5.972941852774387}.
...
[I 2018-05-09 10:40:26,204] Finished a trial resulted in value: 14.704254135013741. Current best value is 2.280758099793617e-06 with parameters: {'x': 1.9984897821018828}.

Please note that foo.py only contains the definition of the objective function. By giving the script file name and the method name of objective function to optuna study optimize command, we can invoke the optimization.

User Attributes¶

This feature is to annotate experiments with user-defined attributes.

Adding User Attributes to Studies¶

A Study object provides set_user_attr() method to register a pair of key and value as an user-defined attribute. A key is supposed to be a str, and a value be any object serializable with json.dumps.

import optuna
study = optuna.create_study(storage='sqlite:///example.db')
study.set_user_attr('contributors', ['Akiba', 'Sano'])
study.set_user_attr('dataset', 'MNIST')

We can access annotated attributes with user_attr property.

study.user_attrs  # {'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'}

StudySummary object, which can be retrieved by get_all_study_summaries(), also contains user-defined attributes.

study_summaries = optuna.get_all_study_summaries('sqlite:///example.db')
study_summaries[0].user_attrs  # {'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'}

See also

optuna study set-user-attr command, which sets an attribute via command line interface.

Adding User Attributes to Trials¶

As with Study, a Trial object provides set_user_attr() method. Attributes are set inside an objective function.

def objective(trial):
    iris = sklearn.datasets.load_iris()
    x, y = iris.data, iris.target

    svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
    clf = sklearn.svm.SVC(C=svc_c)
    accuracy = sklearn.model_selection.cross_val_score(clf, x, y).mean()

    trial.set_user_attr('accuracy', accuracy)

    return 1.0 - accuracy  # return error for minimization

We can access annotated attributes as:

study.trials[0].user_attrs  # {'accuracy': 0.83}

Note that, in this example, the attribute is not annotated to a Study but a single Trial.

Pruning Unpromising Trials¶

This feature automatically stops unpromising trials at the early stages of the training (a.k.a., automated early-stopping). Optuna provides interfaces to concisely implement the pruning mechanism in iterative training algorithms.

Activating Pruners¶

To turn on the pruning feature, you need to call report() and should_prune() after each step of the iterative training. report() periodically monitors the intermediate objective values. should_prune() decides termination of the trial that does not meet a predefined condition.

"""filename: prune.py"""

import sklearn.datasets
import sklearn.linear_model
import sklearn.model_selection

import optuna

def objective(trial):
    iris = sklearn.datasets.load_iris()
    classes = list(set(iris.target))
    train_x, valid_x, train_y, valid_y = \
        sklearn.model_selection.train_test_split(iris.data, iris.target, test_size=0.25, random_state=0)

    alpha = trial.suggest_loguniform('alpha', 1e-5, 1e-1)
    clf = sklearn.linear_model.SGDClassifier(alpha=alpha)

    for step in range(100):
        clf.partial_fit(train_x, train_y, classes=classes)

        # Report intermediate objective value.
        intermediate_value = 1.0 - clf.score(valid_x, valid_y)
        trial.report(intermediate_value, step)

        # Handle pruning based on the intermediate value.
        if trial.should_prune():
            raise optuna.TrialPruned()

    return 1.0 - clf.score(valid_x, valid_y)

# Set up the median stopping rule as the pruning condition.
study = optuna.create_study(pruner=optuna.pruners.MedianPruner())
study.optimize(objective, n_trials=20)

Executing the script above:

$ python prune.py
[I 2020-06-12 16:54:23,876] Trial 0 finished with value: 0.3157894736842105 and parameters: {'alpha': 0.00181467547181131}. Best is trial 0 with value: 0.3157894736842105.
[I 2020-06-12 16:54:23,981] Trial 1 finished with value: 0.07894736842105265 and parameters: {'alpha': 0.015378744419287613}. Best is trial 1 with value: 0.07894736842105265.
[I 2020-06-12 16:54:24,083] Trial 2 finished with value: 0.21052631578947367 and parameters: {'alpha': 0.04089428832878595}. Best is trial 1 with value: 0.07894736842105265.
[I 2020-06-12 16:54:24,185] Trial 3 finished with value: 0.052631578947368474 and parameters: {'alpha': 0.004018735937374473}. Best is trial 3 with value: 0.052631578947368474.
[I 2020-06-12 16:54:24,303] Trial 4 finished with value: 0.07894736842105265 and parameters: {'alpha': 2.805688697062864e-05}. Best is trial 3 with value: 0.052631578947368474.
[I 2020-06-12 16:54:24,315] Trial 5 pruned.
[I 2020-06-12 16:54:24,355] Trial 6 pruned.
[I 2020-06-12 16:54:24,511] Trial 7 finished with value: 0.052631578947368474 and parameters: {'alpha': 2.243775785299103e-05}. Best is trial 3 with value: 0.052631578947368474.
[I 2020-06-12 16:54:24,625] Trial 8 finished with value: 0.1842105263157895 and parameters: {'alpha': 0.007021209286214553}. Best is trial 3 with value: 0.052631578947368474.
[I 2020-06-12 16:54:24,629] Trial 9 pruned.
...

Trial 5 pruned., etc. in the log messages means several trials were stopped before they finished all of the iterations.

Integration Modules for Pruning¶

To implement pruning mechanism in much simpler forms, Optuna provides integration modules for the following libraries.

XGBoost: optuna.integration.XGBoostPruningCallback
LightGBM: optuna.integration.LightGBMPruningCallback
Chainer: optuna.integration.ChainerPruningExtension
Keras: optuna.integration.KerasPruningCallback
TensorFlow optuna.integration.TensorFlowPruningHook
tf.keras optuna.integration.TFKerasPruningCallback
MXNet optuna.integration.MXNetPruningCallback
PyTorch Ignite optuna.integration.PyTorchIgnitePruningHandler
PyTorch Lightning optuna.integration.PyTorchLightningPruningCallback
FastAI optuna.integration.FastAIPruningCallback

For example, XGBoostPruningCallback introduces pruning without directly changing the logic of training iteration. (See also example for the entire script.)

pruning_callback = optuna.integration.XGBoostPruningCallback(trial, 'validation-error')
bst = xgb.train(param, dtrain, evals=[(dvalid, 'validation')], callbacks=[pruning_callback])

User-Defined Sampler¶

Thanks to user-defined samplers, you can:

experiment your own sampling algorithms,
implement task-specific algorithms to refine the optimization performance, or
wrap other optimization libraries to integrate them into Optuna pipelines (e.g., SkoptSampler).

This section describes the internal behavior of sampler classes and shows an example of implementing a user-defined sampler.

Overview of Sampler¶

A sampler has the responsibility to determine the parameter values to be evaluated in a trial. When a suggest API (e.g., suggest_uniform()) is called inside an objective function, the corresponding distribution object (e.g., UniformDistribution) is created internally. A sampler samples a parameter value from the distribution. The sampled value is returned to the caller of the suggest API and evaluated in the objective function.

To create a new sampler, you need to define a class that inherits BaseSampler. The base class has three abstract methods; infer_relative_search_space(), sample_relative(), and sample_independent().

As the method names imply, Optuna supports two types of sampling: one is relative sampling that can consider the correlation of the parameters in a trial, and the other is independent sampling that samples each parameter independently.

At the beginning of a trial, infer_relative_search_space() is called to provide the relative search space for the trial. Then, sample_relative() is invoked to sample relative parameters from the search space. During the execution of the objective function, sample_independent() is used to sample parameters that don’t belong to the relative search space.

Note

Please refer to the document of BaseSampler for further details.

An Example: Implementing SimulatedAnnealingSampler¶

For example, the following code defines a sampler based on Simulated Annealing (SA):

import numpy as np
import optuna


class SimulatedAnnealingSampler(optuna.samplers.BaseSampler):
    def __init__(self, temperature=100):
        self._rng = np.random.RandomState()
        self._temperature = temperature  # Current temperature.
        self._current_trial = None  # Current state.

    def sample_relative(self, study, trial, search_space):
        if search_space == {}:
            return {}

        #
        # An implementation of SA algorithm.
        #

        # Calculate transition probability.
        prev_trial = study.trials[-2]
        if self._current_trial is None or prev_trial.value <= self._current_trial.value:
            probability = 1.0
        else:
            probability = np.exp((self._current_trial.value - prev_trial.value) / self._temperature)
        self._temperature *= 0.9  # Decrease temperature.

        # Transit the current state if the previous result is accepted.
        if self._rng.uniform(0, 1) < probability:
            self._current_trial = prev_trial

        # Sample parameters from the neighborhood of the current point.
        #
        # The sampled parameters will be used during the next execution of
        # the objective function passed to the study.
        params = {}
        for param_name, param_distribution in search_space.items():
            if not isinstance(param_distribution, optuna.distributions.UniformDistribution):
                raise NotImplementedError('Only suggest_uniform() is supported')

            current_value = self._current_trial.params[param_name]
            width = (param_distribution.high - param_distribution.low) * 0.1
            neighbor_low = max(current_value - width, param_distribution.low)
            neighbor_high = min(current_value + width, param_distribution.high)
            params[param_name] = self._rng.uniform(neighbor_low, neighbor_high)

        return params

    #
    # The rest is boilerplate code and unrelated to SA algorithm.
    #
    def infer_relative_search_space(self, study, trial):
        return optuna.samplers.intersection_search_space(study)

    def sample_independent(self, study, trial, param_name, param_distribution):
        independent_sampler = optuna.samplers.RandomSampler()
        return independent_sampler.sample_independent(study, trial, param_name, param_distribution)

Note

In favor of code simplicity, the above implementation doesn’t support some features (e.g., maximization). If you’re interested in how to support those features, please see examples/samplers/simulated_annealing.py.

You can use SimulatedAnnealingSampler in the same way as built-in samplers as follows:

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    y = trial.suggest_uniform('y', -5, 5)
    return x**2 + y

sampler = SimulatedAnnealingSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=100)

In this optimization, the values of x and y parameters are sampled by using SimulatedAnnealingSampler.sample_relative method.

Note

Strictly speaking, in the first trial, SimulatedAnnealingSampler.sample_independent method is used to sample parameter values. Because intersection_search_space() used in SimulatedAnnealingSampler.infer_relative_search_space cannot infer the search space if there are no complete trials.

API Reference¶

optuna¶

optuna.create_study(storage=None, sampler=None, pruner=None, study_name=None, direction='minimize', load_if_exists=False)[source]¶

Create a new Study.

Parameters

storage –
Database URL. If this argument is set to None, in-memory storage is used, and the Study will not be persistent.

Note

When a database URL is passed, Optuna internally uses SQLAlchemy to handle the database. Please refer to SQLAlchemy’s document for further details. If you want to specify non-default options to SQLAlchemy Engine, you can instantiate RDBStorage with your desired options and pass it to the storage argument instead of a URL.
sampler – A sampler object that implements background algorithm for value suggestion. If None is specified, TPESampler is used as the default. See also samplers.
pruner – A pruner object that decides early stopping of unpromising trials. See also pruners.
study_name – Study’s name. If this argument is set to None, a unique name is generated automatically.
direction – Direction of optimization. Set minimize for minimization and maximize for maximization.
load_if_exists – Flag to control the behavior to handle a conflict of study names. In the case where a study named study_name already exists in the storage, a DuplicatedStudyError is raised if load_if_exists is set to False. Otherwise, the creation of the study is skipped, and the existing one is returned.

Returns

A Study object.

See also

This is an alias for optuna.study.create_study().

optuna.load_study(study_name, storage, sampler=None, pruner=None)[source]¶

Load the existing Study that has the specified name.

Parameters

study_name – Study’s name. Each study has a unique name as an identifier.
storage – Database URL such as sqlite:///example.db. Please see also the documentation of create_study() for further details.
sampler – A sampler object that implements background algorithm for value suggestion. If None is specified, TPESampler is used as the default. See also samplers.
pruner – A pruner object that decides early stopping of unpromising trials. If None is specified, MedianPruner is used as the default. See also pruners.

See also

This is an alias for optuna.study.load_study().

optuna.delete_study(study_name, storage)[source]¶

Delete a Study object.

Parameters

study_name – Study’s name.
storage – Database URL such as sqlite:///example.db. Please see also the documentation of create_study() for further details.

See also

This is an alias for optuna.study.delete_study().

optuna.get_all_study_summaries(storage)[source]¶

Get all history of studies stored in a specified storage.

Parameters: storage – Database URL such as sqlite:///example.db. Please see also the documentation of create_study() for further details.
Returns: List of study history summarized as StudySummary objects.

See also

This is an alias for optuna.study.get_all_study_summaries().

class optuna.TrialPruned[source]¶

Exception for pruned trials.

This error tells a trainer that the current Trial was pruned. It is supposed to be raised after optuna.trial.Trial.should_prune() as shown in the following example.

Example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)
classes = np.unique(y)

def objective(trial):
    alpha = trial.suggest_uniform('alpha', 0.0, 1.0)
    clf = SGDClassifier(alpha=alpha)
    n_train_iter = 100

    for step in range(n_train_iter):
        clf.partial_fit(X_train, y_train, classes=classes)

        intermediate_value = clf.score(X_valid, y_valid)
        trial.report(intermediate_value, step)

        if trial.should_prune():
            raise optuna.TrialPruned()

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=20)

See also

This is an alias for optuna.exceptions.TrialPruned.

optuna.cli¶

optuna
    [--version]
    [-v | -q]
    [--log-file LOG_FILE]
    [--debug]
    [--storage STORAGE]

--version¶: show program’s version number and exit

-v, --verbose¶: Increase verbosity of output. Can be repeated.

-q, --quiet¶: Suppress output except warnings and errors.

--log-file <LOG_FILE>¶: Specify a file to log output. Disabled by default.

--debug¶: Show tracebacks on errors.

--storage <STORAGE>¶: DB URL. (e.g. sqlite:///example.db)

create-study¶

Create a new study.

optuna create-study
    [--study-name STUDY_NAME]
    [--direction {minimize,maximize}]
    [--skip-if-exists]

--study-name <STUDY_NAME>¶: A human-readable name of a study to distinguish it from others.

--direction <DIRECTION>¶: Set direction of optimization to a new study. Set ‘minimize’ for minimization and ‘maximize’ for maximization.

--skip-if-exists¶: If specified, the creation of the study is skipped without any error when the study name is duplicated.

This command is provided by the optuna plugin.

dashboard¶

Launch web dashboard (beta).

optuna dashboard
    [--study STUDY]
    [--study-name STUDY_NAME]
    [--out OUT]
    [--allow-websocket-origin BOKEH_ALLOW_WEBSOCKET_ORIGINS]

--study <STUDY>¶: This argument is deprecated. Use –study-name instead.

--study-name <STUDY_NAME>¶: A human-readable name of a study to distinguish it from others.

--out <OUT>, -o <OUT>¶: Output HTML file path. If it is not given, a HTTP server starts and the dashboard is served.

--allow-websocket-origin <BOKEH_ALLOW_WEBSOCKET_ORIGINS>¶: Allow websocket access from the specified host(s).Internally, it is used as the value of bokeh’s –allow-websocket-origin option. Please refer to https://bokeh.pydata.org/en/latest/docs/reference/command/subcommands/serve.html for more details.

This command is provided by the optuna plugin.

delete-study¶

Delete a specified study.

optuna delete-study [--study-name STUDY_NAME]

--study-name <STUDY_NAME>¶: A human-readable name of a study to distinguish it from others.

This command is provided by the optuna plugin.

storage upgrade¶

Upgrade the schema of a storage.

optuna storage upgrade

This command is provided by the optuna plugin.

studies¶

Show a list of studies.

optuna studies
    [-f {csv,json,table,value,yaml}]
    [-c COLUMN]
    [--quote {all,minimal,none,nonnumeric}]
    [--noindent]
    [--max-width <integer>]
    [--fit-width]
    [--print-empty]
    [--sort-column SORT_COLUMN]

-f <FORMATTER>, --format <FORMATTER>¶: the output format, defaults to table

-c COLUMN, --column COLUMN¶: specify the column(s) to include, can be repeated to show multiple columns

--quote <QUOTE_MODE>¶: when to include quotes, defaults to nonnumeric

--noindent¶: whether to disable indenting the JSON

--max-width <integer>¶: Maximum display width, <1 to disable. You can also use the CLIFF_MAX_TERM_WIDTH environment variable, but the parameter takes precedence.

--fit-width¶: Fit the table to the display width. Implied if –max-width greater than 0. Set the environment variable CLIFF_FIT_WIDTH=1 to always enable

--print-empty¶: Print empty table if there is no data to show.

--sort-column SORT_COLUMN¶: specify the column(s) to sort the data (columns specified first have a priority, non-existing columns are ignored), can be repeated

This command is provided by the optuna plugin.

study optimize¶

Start optimization of a study. Deprecated since version 2.0.0.

optuna study optimize
    [--n-trials N_TRIALS]
    [--timeout TIMEOUT]
    [--n-jobs N_JOBS]
    [--study STUDY]
    [--study-name STUDY_NAME]
    file
    method

--n-trials <N_TRIALS>¶: The number of trials. If this argument is not given, as many trials run as possible.

--timeout <TIMEOUT>¶: Stop study after the given number of second(s). If this argument is not given, as many trials run as possible.

--n-jobs <N_JOBS>¶: The number of parallel jobs. If this argument is set to -1, the number is set to CPU counts.

--study <STUDY>¶: This argument is deprecated. Use –study-name instead.

--study-name <STUDY_NAME>¶: A human-readable name of a study to distinguish it from others.

file¶: Python script file where the objective function resides.

method¶: The method name of the objective function.

This command is provided by the optuna plugin.

study set-user-attr¶

Set a user attribute to a study.

optuna study set-user-attr
    [--study STUDY]
    [--study-name STUDY_NAME]
    --key KEY
    --value VALUE

--study <STUDY>¶: This argument is deprecated. Use –study-name instead.

--study-name <STUDY_NAME>¶: A human-readable name of a study to distinguish it from others.

--key <KEY>, -k <KEY>¶: Key of the user attribute.

--value <VALUE>, -v <VALUE>¶: Value to be set.

This command is provided by the optuna plugin.

optuna.distributions¶

class optuna.distributions.UniformDistribution(low, high)[source]¶

A uniform distribution in the linear domain.

This object is instantiated by suggest_uniform(), and passed to samplers in general.

low¶: Lower endpoint of the range of the distribution. low is included in the range.

high¶: Upper endpoint of the range of the distribution. high is excluded from the range.

single()[source]¶

Test whether the range of this distribution contains just a single value.

When this method returns True, samplers always sample the same value from the distribution.

Returns: True if the range of this distribution contains just a single value, otherwise False.

class optuna.distributions.LogUniformDistribution(low, high)[source]¶

A uniform distribution in the log domain.

This object is instantiated by suggest_loguniform(), and passed to samplers in general.

low¶: Lower endpoint of the range of the distribution. low is included in the range.

high¶: Upper endpoint of the range of the distribution. high is excluded from the range.

single()[source]¶

Test whether the range of this distribution contains just a single value.

When this method returns True, samplers always sample the same value from the distribution.

Returns: True if the range of this distribution contains just a single value, otherwise False.

class optuna.distributions.DiscreteUniformDistribution(low: float, high: float, q: float)[source]¶

A discretized uniform distribution in the linear domain.

This object is instantiated by suggest_discrete_uniform(), and passed to samplers in general.

Note

If the range $[\mathsf{low}, \mathsf{high}]$ is not divisible by $q$, $\mathsf{high}$ will be replaced with the maximum of $k q + \mathsf{low} \lt \mathsf{high}$, where $k$ is an integer.

low¶: Lower endpoint of the range of the distribution. low is included in the range.

high¶: Upper endpoint of the range of the distribution. high is included in the range.

q¶: A discretization step.

single()[source]¶

Test whether the range of this distribution contains just a single value.

When this method returns True, samplers always sample the same value from the distribution.

Returns: True if the range of this distribution contains just a single value, otherwise False.

class optuna.distributions.IntUniformDistribution(low: int, high: int, step: int = 1)[source]¶

A uniform distribution on integers.

This object is instantiated by suggest_int(), and passed to samplers in general.

Note

If the range $[\mathsf{low}, \mathsf{high}]$ is not divisible by $\mathsf{step}$, $\mathsf{high}$ will be replaced with the maximum of $k \times \mathsf{step} + \mathsf{low} \lt \mathsf{high}$, where $k$ is an integer.

low¶: Lower endpoint of the range of the distribution. low is included in the range.

high¶: Upper endpoint of the range of the distribution. high is included in the range.

step¶: A step for spacing between values.

single()[source]¶

Test whether the range of this distribution contains just a single value.

When this method returns True, samplers always sample the same value from the distribution.

Returns: True if the range of this distribution contains just a single value, otherwise False.

class optuna.distributions.IntLogUniformDistribution(low: int, high: int, step: int = 1)[source]¶

A uniform distribution on integers in the log domain.

This object is instantiated by suggest_int(), and passed to samplers in general.

Note

If the range $[\mathsf{low}, \mathsf{high}]$ is not divisible by $\mathsf{step}$, $\mathsf{high}$ will be replaced with the maximum of $k \times \mathsf{step} + \mathsf{low} \lt \mathsf{high}$, where $k$ is an integer.

low¶: Lower endpoint of the range of the distribution. low is included in the range.

high¶: Upper endpoint of the range of the distribution. high is included in the range.

step¶: A step for spacing between values.

single()[source]¶

Test whether the range of this distribution contains just a single value.

When this method returns True, samplers always sample the same value from the distribution.

Returns: True if the range of this distribution contains just a single value, otherwise False.

class optuna.distributions.CategoricalDistribution(choices)[source]¶

A categorical distribution.

This object is instantiated by suggest_categorical(), and passed to samplers in general.

Parameters: choices – Parameter value candidates.

Note

Not all types are guaranteed to be compatible with all storages. It is recommended to restrict the types of the choices to None, bool, int, float and str.

choices¶: Parameter value candidates.

single()[source]¶

Test whether the range of this distribution contains just a single value.

When this method returns True, samplers always sample the same value from the distribution.

Returns: True if the range of this distribution contains just a single value, otherwise False.

optuna.distributions.distribution_to_json(dist)[source]¶

Serialize a distribution to JSON format.

Parameters: dist – A distribution to be serialized.
Returns: A JSON string of a given distribution.

optuna.distributions.json_to_distribution(json_str)[source]¶

Deserialize a distribution in JSON format.

Parameters: json_str – A JSON-serialized distribution.
Returns: A deserialized distribution.

optuna.distributions.check_distribution_compatibility(dist_old, dist_new)[source]¶

A function to check compatibility of two distributions.

Note that this method is not supposed to be called by library users.

Parameters

dist_old – A distribution previously recorded in storage.
dist_new – A distribution newly added to storage.

Returns

True denotes given distributions are compatible. Otherwise, they are not.

optuna.exceptions¶

class optuna.exceptions.OptunaError[source]¶: Base class for Optuna specific errors.

class optuna.exceptions.TrialPruned[source]¶

Exception for pruned trials.

This error tells a trainer that the current Trial was pruned. It is supposed to be raised after optuna.trial.Trial.should_prune() as shown in the following example.

Example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)
classes = np.unique(y)

def objective(trial):
    alpha = trial.suggest_uniform('alpha', 0.0, 1.0)
    clf = SGDClassifier(alpha=alpha)
    n_train_iter = 100

    for step in range(n_train_iter):
        clf.partial_fit(X_train, y_train, classes=classes)

        intermediate_value = clf.score(X_valid, y_valid)
        trial.report(intermediate_value, step)

        if trial.should_prune():
            raise optuna.TrialPruned()

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=20)

See also

The alias also exists as optuna.TrialPruned.

class optuna.exceptions.CLIUsageError[source]¶

Exception for CLI.

CLI raises this exception when it receives invalid configuration.

class optuna.exceptions.StorageInternalError[source]¶

Exception for storage operation.

This error is raised when an operation failed in backend DB of storage.

class optuna.exceptions.DuplicatedStudyError[source]¶

Exception for a duplicated study name.

This error is raised when a specified study name already exists in the storage.

optuna.importance¶

optuna.importance.get_param_importances(study: optuna.study.Study, evaluator: optuna.importance._base.BaseImportanceEvaluator = None, params: Optional[List[str]] = None) → Dict[str, float][source]¶

Evaluate parameter importances based on completed trials in the given study.

The parameter importances are returned as a dictionary where the keys consist of parameter names and their values importances. The importances are represented by floating point numbers that sum to 1.0 over the entire dictionary. The higher the value, the more important. The returned dictionary is of type collections.OrderedDict and is ordered by its values in a descending order.

If params is None, all parameter that are present in all of the completed trials are assessed. This implies that conditional parameters will be excluded from the evaluation. To assess the importances of conditional parameters, a list of parameter names can be specified via params. If specified, only completed trials that contain all of the parameters will be considered. If no such trials are found, an error will be raised.

If the given study does not contain completed trials, an error will be raised.

Note

If params is specified as an empty list, an empty dictionary is returned.

See also

See plot_param_importances() to plot importances.

Parameters

study – An optimized study.
evaluator – An importance evaluator object that specifies which algorithm to base the importance assessment on. Defaults to MeanDecreaseImpurityImportanceEvaluator.
params – A list of names of parameters to assess. If None, all parameters that are present in all of the completed trials are assessed.

Returns

An collections.OrderedDict where the keys are parameter names and the values are assessed importances.

Note

Added in v1.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.3.0.

class optuna.importance.FanovaImportanceEvaluator[source]¶

fANOVA parameter importance evaluator.

Note

Requires the fanova Python package.

Note

Added in v1.3.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.3.0.

evaluate(study: optuna.study.Study, params: Optional[List[str]]) → Dict[str, float][source]¶

Evaluate parameter importances based on completed trials in the given study.

Note

This method is not meant to be called by library users.

See also

Please refer to get_param_importances() for how a concrete evaluator should implement this method.

Parameters

study – An optimized study.
params – A list of names of parameters to assess. If None, all parameters that are present in all of the completed trials are assessed.

Returns

An collections.OrderedDict where the keys are parameter names and the values are assessed importances.

class optuna.importance.MeanDecreaseImpurityImportanceEvaluator(n_estimators: int = 16, max_depth: int = 64, random_state: Optional[int] = None)[source]¶

Mean Decrease Impurity (MDI) parameter importance evaluator.

This evaluator fits a random forest that predicts objective values given hyperparameter configurations. Feature importances are then computed using MDI.

Note

This evaluator requires the sklean Python package and is based on sklearn.ensemble.RandomForestClassifier.feature_importances_.

Parameters

n_estimators – Number of trees in the random forest.
max_depth – The maximum depth of each tree in the random forest.
random_seed – Seed for the random forest.

Note

Added in v1.5.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.5.0.

evaluate(study: optuna.study.Study, params: Optional[List[str]]) → Dict[str, float][source]¶

Evaluate parameter importances based on completed trials in the given study.

Note

This method is not meant to be called by library users.

See also

Please refer to get_param_importances() for how a concrete evaluator should implement this method.

Parameters

study – An optimized study.
params – A list of names of parameters to assess. If None, all parameters that are present in all of the completed trials are assessed.

Returns

An collections.OrderedDict where the keys are parameter names and the values are assessed importances.

optuna.integration¶

class optuna.integration.ChainerPruningExtension(trial, observation_key, pruner_trigger)[source]¶

Chainer extension to prune unpromising trials.

See the example if you want to add a pruning extension which observes validation accuracy of a Chainer Trainer.

Parameters

trial – A Trial corresponding to the current evaluation of the objective function.
observation_key – An evaluation metric for pruning, e.g., main/loss and validation/main/accuracy. Please refer to chainer.Reporter reference for further details.
pruner_trigger –
A trigger to execute pruning. pruner_trigger is an instance of IntervalTrigger or ManualScheduleTrigger. IntervalTrigger can be specified by a tuple of the interval length and its unit like (1, 'epoch').

class optuna.integration.ChainerMNStudy(study, comm)[source]¶

A wrapper of Study to incorporate Optuna with ChainerMN.

See also

ChainerMNStudy provides the same interface as Study. Please refer to optuna.study.Study for further details.

See the example if you want to optimize an objective function that trains neural network written with ChainerMN.

Parameters

study – A Study object.
comm – A ChainerMN communicator.

optimize(func, n_trials=None, timeout=None, catch=())[source]¶

Optimize an objective function.

This method provides the same interface as optuna.study.Study.optimize() except the absence of n_jobs argument.

class optuna.integration.PyCmaSampler(x0=None, sigma0=None, cma_stds=None, seed=None, cma_opts=None, n_startup_trials=1, independent_sampler=None, warn_independent_sampling=True)[source]¶

A Sampler using cma library as the backend.

Example

Optimize a simple quadratic function by using PyCmaSampler.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -1, 1)
    y = trial.suggest_int('y', -1, 1)
    return x**2 + y

sampler = optuna.integration.PyCmaSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=20)

Note that parallel execution of trials may affect the optimization performance of CMA-ES, especially if the number of trials running in parallel exceeds the population size.

Note

CmaEsSampler is deprecated and renamed to PyCmaSampler in v2.0.0. Please use PyCmaSampler instead of CmaEsSampler.

Parameters

x0 – A dictionary of an initial parameter values for CMA-ES. By default, the mean of low and high for each distribution is used. Please refer to cma.CMAEvolutionStrategy for further details of x0.
sigma0 – Initial standard deviation of CMA-ES. By default, sigma0 is set to min_range / 6, where min_range denotes the minimum range of the distributions in the search space. If distribution is categorical, min_range is len(choices) - 1. Please refer to cma.CMAEvolutionStrategy for further details of sigma0.
cma_stds – A dictionary of multipliers of sigma0 for each parameters. The default value is 1.0. Please refer to cma.CMAEvolutionStrategy for further details of cma_stds.
seed – A random seed for CMA-ES.
cma_opts –
Options passed to the constructor of cma.CMAEvolutionStrategy class.

Note that BoundaryHandler, bounds, CMA_stds and seed arguments in cma_opts will be ignored because it is added by PyCmaSampler automatically.
n_startup_trials – The independent sampling is used instead of the CMA-ES algorithm until the given number of trials finish in the same study.
independent_sampler –
A BaseSampler instance that is used for independent sampling. The parameters not contained in the relative search space are sampled by this sampler. The search space for PyCmaSampler is determined by intersection_search_space().

If None is specified, RandomSampler is used as the default.

See also

optuna.samplers module provides built-in independent samplers such as RandomSampler and TPESampler.
warn_independent_sampling –
If this is True, a warning message is emitted when the value of a parameter is sampled by using an independent sampler.

Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case.

reseed_rng() → None [source]¶

Reseed sampler’s random number generator.

This method is called by the Study instance if trials are executed in parallel with the option n_jobs>1. In that case, the sampler instance will be replicated including the state of the random number generator, and they may suggest the same values. To prevent this issue, this method assigns a different seed to each random number generator.

class optuna.integration.CmaEsSampler(x0=None, sigma0=None, cma_stds=None, seed=None, cma_opts=None, n_startup_trials=1, independent_sampler=None, warn_independent_sampling=True)[source]¶: Wrapper class of PyCmaSampler for backward compatibility.

Deprecated since version 2.0.0: This class is renamed to PyCmaSampler.

class optuna.integration.FastAIPruningCallback(learn, trial, monitor)[source]¶

FastAI callback to prune unpromising trials for fastai.

Note

This callback is for fastai<2.0, not the coming version developed in fastai/fastai_dev.

See the example if you want to add a pruning callback which monitors validation loss of a Learner.

Example

Register a pruning callback to learn.fit and learn.fit_one_cycle.

learn.fit(n_epochs, callbacks=[FastAIPruningCallback(learn, trial, 'valid_loss')])
learn.fit_one_cycle(
    n_epochs, cyc_len, max_lr,
    callbacks=[FastAIPruningCallback(learn, trial, 'valid_loss')])

Parameters

learn – fastai.basic_train.Learner.
trial – A Trial corresponding to the current evaluation of the objective function.
monitor – An evaluation metric for pruning, e.g. valid_loss and Accuracy. Please refer to fastai.Callback reference for further details.

class optuna.integration.PyTorchIgnitePruningHandler(trial, metric, trainer)[source]¶

PyTorch Ignite handler to prune unpromising trials.

See the example if you want to add a pruning handler which observes validation accuracy.

Parameters

trial – A Trial corresponding to the current evaluation of the objective function.
metric – A name of metric for pruning, e.g., accuracy and loss.
trainer – A trainer engine of PyTorch Ignite. Please refer to ignite.engine.Engine reference for further details.

class optuna.integration.KerasPruningCallback(trial, monitor, interval=1)[source]¶

Keras callback to prune unpromising trials.

See the example if you want to add a pruning callback which observes validation accuracy.

Parameters

trial – A Trial corresponding to the current evaluation of the objective function.
monitor – An evaluation metric for pruning, e.g., val_loss and val_accuracy. Please refer to keras.Callback reference for further details.
interval – Check if trial should be pruned every n-th epoch. By default interval=1 and pruning is performed after every epoch. Increase interval to run several epochs faster before applying pruning.

class optuna.integration.LightGBMPruningCallback(trial, metric, valid_name='valid_0')[source]¶

Callback for LightGBM to prune unpromising trials.

See the example if you want to add a pruning callback which observes AUC of a LightGBM model.

Parameters

trial – A Trial corresponding to the current evaluation of the objective function.
metric – An evaluation metric for pruning, e.g., binary_error and multi_error. Please refer to LightGBM reference for further details.
valid_name – The name of the target validation. Validation names are specified by valid_names option of train method. If omitted, valid_0 is used which is the default name of the first validation. Note that this argument will be ignored if you are calling cv method instead of train method.

optuna.integration.lightgbm.train(*args: Any, **kwargs: Any) → Any[source]¶

Wrapper of LightGBM Training API to tune hyperparameters.

It tunes important hyperparameters (e.g., min_child_samples and feature_fraction) in a stepwise manner. It is a drop-in replacement for lightgbm.train(). See a simple example of LightGBM Tuner which optimizes the validation log loss of cancer detection.

train() is a wrapper function of LightGBMTuner. To use feature in Optuna such as suspended/resumed optimization and/or parallelization, refer to LightGBMTuner instead of this function.

Arguments and keyword arguments for lightgbm.train() can be passed.

Note

Added in v0.18.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v0.18.0.

class optuna.integration.lightgbm.LightGBMTuner(params: Dict[str, Any], train_set: lgb.Dataset, num_boost_round: int = 1000, valid_sets: Optional[VALID_SET_TYPE] = None, valid_names: Optional[Any] = None, fobj: Optional[Callable[[…], Any]] = None, feval: Optional[Callable[[…], Any]] = None, feature_name: str = 'auto', categorical_feature: str = 'auto', early_stopping_rounds: Optional[int] = None, evals_result: Optional[Dict[Any, Any]] = None, verbose_eval: Union[bool, int, None] = True, learning_rates: Optional[List[float]] = None, keep_training_booster: Optional[bool] = False, callbacks: Optional[List[Callable[[…], Any]]] = None, time_budget: Optional[int] = None, sample_size: Optional[int] = None, study: Optional[optuna.study.Study] = None, optuna_callbacks: Optional[List[Callable[[optuna.study.Study, optuna.trial._frozen.FrozenTrial], None]]] = None, model_dir: Optional[str] = None, verbosity: Optional[int] = 1)[source]¶

Hyperparameter tuner for LightGBM.

It optimizes the following hyperparameters in a stepwise manner: lambda_l1, lambda_l2, num_leaves, feature_fraction, bagging_fraction, bagging_freq and min_child_samples.

You can find the details of the algorithm and benchmark results in this blog article by Kohei Ozaki, a Kaggle Grandmaster.

Arguments and keyword arguments for lightgbm.train() can be passed. The arguments that only LightGBMTuner has are listed below:

Parameters

time_budget – A time budget for parameter tuning in seconds.
study – A Study instance to store optimization results. The Trial instances in it has the following user attributes: elapsed_secs is the elapsed time since the optimization starts. average_iteration_time is the average time of iteration to train the booster model in the trial. lgbm_params is a JSON-serialized dictionary of LightGBM parameters used in the trial.
optuna_callbacks – List of Optuna callback functions that are invoked at the end of each trial. Each function must accept two parameters with the following types in this order: Study and FrozenTrial. Please note that this is not a callbacks argument of lightgbm.train() .
model_dir – A directory to save boosters. By default, it is set to None and no boosters are saved. Please set shared directory (e.g., directories on NFS) if you want to access get_best_booster() in distributed environments. Otherwise, it may raise ValueError. If the directory does not exist, it will be created. The filenames of the boosters will be {model_dir}/{trial_number}.pkl (e.g., ./boosters/0.pkl).

Note

Added in v1.5.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.5.0.

property best_booster¶: Return the best booster.

Deprecated since version 1.4.0: Please get the best booster via get_best_booster instead.

property best_params¶: Return parameters of the best booster.

property best_score¶: Return the score of the best booster.

get_best_booster() → lgb.Booster[source]¶

Return the best booster.

If the best booster cannot be found, ValueError will be raised. To prevent the errors, please save boosters by specifying the model_dir arguments of __init__() when you resume tuning or you run tuning in parallel.

run() → None ¶: Perform the hyperparameter-tuning with given parameters.

class optuna.integration.lightgbm.LightGBMTunerCV(params: Dict[str, Any], train_set: lgb.Dataset, num_boost_round: int = 1000, folds: Union[Generator[Tuple[int, int], None, None], Iterator[Tuple[int, int]], BaseCrossValidator, None] = None, nfold: int = 5, stratified: bool = True, shuffle: bool = True, fobj: Optional[Callable[[…], Any]] = None, feval: Optional[Callable[[…], Any]] = None, feature_name: str = 'auto', categorical_feature: str = 'auto', early_stopping_rounds: Optional[int] = None, fpreproc: Optional[Callable[[…], Any]] = None, verbose_eval: Union[bool, int, None] = True, show_stdv: bool = True, seed: int = 0, callbacks: Optional[List[Callable[[…], Any]]] = None, time_budget: Optional[int] = None, sample_size: Optional[int] = None, study: Optional[optuna.study.Study] = None, optuna_callbacks: Optional[List[Callable[[optuna.study.Study, optuna.trial._frozen.FrozenTrial], None]]] = None, verbosity: int = 1)[source]¶

Hyperparameter tuner for LightGBM with cross-validation.

It employs the same stepwise approach as LightGBMTuner. LightGBMTunerCV invokes lightgbm.cv() to train and validate boosters while LightGBMTuner invokes lightgbm.train(). See a simple example which optimizes the validation log loss of cancer detection.

Arguments and keyword arguments for lightgbm.cv() can be passed except metrics, init_model and eval_train_metric. The arguments that only LightGBMTunerCV has are listed below:

Parameters

time_budget – A time budget for parameter tuning in seconds.
study – A Study instance to store optimization results. The Trial instances in it has the following user attributes: elapsed_secs is the elapsed time since the optimization starts. average_iteration_time is the average time of iteration to train the booster model in the trial. lgbm_params is a JSON-serialized dictionary of LightGBM parameters used in the trial.
optuna_callbacks – List of Optuna callback functions that are invoked at the end of each trial. Each function must accept two parameters with the following types in this order: Study and FrozenTrial. Please note that this is not a callbacks argument of lightgbm.train() .

Note

Added in v1.5.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.5.0.

property best_params¶: Return parameters of the best booster.

property best_score¶: Return the score of the best booster.

run() → None ¶: Perform the hyperparameter-tuning with given parameters.

class optuna.integration.MLflowCallback(tracking_uri=None, metric_name='value')[source]¶

Callback to track Optuna trials with MLflow.

This callback adds relevant information that is tracked by Optuna to MLflow. The MLflow experiment will be named after the Optuna study name.

Example

Add MLflow callback to Optuna optimization.

import optuna
from optuna.integration.mlflow import MLflowCallback

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

mlflc = MLflowCallback(
    tracking_uri=YOUR_TRACKING_URI,
    metric_name='my metric score',
)

study = optuna.create_study(study_name='my_study')
study.optimize(objective, n_trials=10, callbacks=[mlflc])

Parameters

tracking_uri –
The URI of the MLflow tracking server.

Please refer to mlflow.set_tracking_uri for more details.
metric_name – Name of the metric. Since the metric itself is just a number, metric_name can be used to give it a name. So you know later if it was roc-auc or accuracy.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

class optuna.integration.MXNetPruningCallback(trial, eval_metric)[source]¶

MXNet callback to prune unpromising trials.

See the example if you want to add a pruning callback which observes accuracy.

Parameters

trial – A Trial corresponding to the current evaluation of the objective function.
eval_metric – An evaluation metric name for pruning, e.g., cross-entropy and accuracy. If using default metrics like mxnet.metrics.Accuracy, use it’s default metric name. For custom metrics, use the metric_name provided to constructor. Please refer to mxnet.metrics reference for further details.

class optuna.integration.PyTorchLightningPruningCallback(trial: optuna.trial._trial.Trial, monitor: str)[source]¶

PyTorch Lightning callback to prune unpromising trials.

See the example if you want to add a pruning callback which observes accuracy.

Parameters

trial – A Trial corresponding to the current evaluation of the objective function.
monitor – An evaluation metric for pruning, e.g., val_loss or val_acc. The metrics are obtained from the returned dictionaries from e.g. pytorch_lightning.LightningModule.training_step or pytorch_lightning.LightningModule.validation_end and the names thus depend on how this dictionary is formatted.

class optuna.integration.SkoptSampler(independent_sampler=None, warn_independent_sampling=True, skopt_kwargs=None, n_startup_trials=1)[source]¶

Sampler using Scikit-Optimize as the backend.

Example

Optimize a simple quadratic function by using SkoptSampler.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    y = trial.suggest_int('y', 0, 10)
    return x**2 + y

sampler = optuna.integration.SkoptSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=10)

Parameters

independent_sampler –
A BaseSampler instance that is used for independent sampling. The parameters not contained in the relative search space are sampled by this sampler. The search space for SkoptSampler is determined by intersection_search_space().

If None is specified, RandomSampler is used as the default.

See also

optuna.samplers module provides built-in independent samplers such as RandomSampler and TPESampler.
warn_independent_sampling –
If this is True, a warning message is emitted when the value of a parameter is sampled by using an independent sampler.

Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case.
skopt_kwargs –
Keyword arguments passed to the constructor of skopt.Optimizer class.

Note that dimensions argument in skopt_kwargs will be ignored because it is added by SkoptSampler automatically.
n_startup_trials – The independent sampling is used until the given number of trials finish in the same study.

reseed_rng() → None [source]¶

Reseed sampler’s random number generator.

This method is called by the Study instance if trials are executed in parallel with the option n_jobs>1. In that case, the sampler instance will be replicated including the state of the random number generator, and they may suggest the same values. To prevent this issue, this method assigns a different seed to each random number generator.

class optuna.integration.TensorFlowPruningHook(trial, estimator, metric, run_every_steps)[source]¶

TensorFlow SessionRunHook to prune unpromising trials.

See the example if you want to add a pruning hook to TensorFlow’s estimator.

Parameters

trial – A Trial corresponding to the current evaluation of the objective function.
estimator – An estimator which you will use.
metric – An evaluation metric for pruning, e.g., accuracy and loss.
run_every_steps – An interval to watch the summary file.

class optuna.integration.TFKerasPruningCallback(trial, monitor)[source]¶

tf.keras callback to prune unpromising trials.

This callback is intend to be compatible for TensorFlow v1 and v2, but only tested with TensorFlow v1.

See the example if you want to add a pruning callback which observes the validation accuracy.

Parameters

trial – A Trial corresponding to the current evaluation of the objective function.
monitor – An evaluation metric for pruning, e.g., val_loss or val_acc.

class optuna.integration.XGBoostPruningCallback(trial, observation_key)[source]¶

Callback for XGBoost to prune unpromising trials.

See the example if you want to add a pruning callback which observes validation AUC of a XGBoost model.

Parameters

trial – A Trial corresponding to the current evaluation of the objective function.
observation_key – An evaluation metric for pruning, e.g., validation-error and validation-merror. When using the Scikit-Learn API, the index number of eval_set must be included in the observation_key, e.g., validation_0-error and validation_0-merror. Please refer to eval_metric in XGBoost reference for further details.

class optuna.integration.OptunaSearchCV(estimator, param_distributions, cv=5, enable_pruning=False, error_score=nan, max_iter=1000, n_jobs=1, n_trials=10, random_state=None, refit=True, return_train_score=False, scoring=None, study=None, subsample=1.0, timeout=None, verbose=0)[source]¶

Hyperparameter search with cross-validation.

Warning

This feature is experimental. The interface may be changed in the future.

Parameters

estimator – Object to use to fit the data. This is assumed to implement the scikit-learn estimator interface. Either this needs to provide score, or scoring must be passed.
param_distributions – Dictionary where keys are parameters and values are distributions. Distributions are assumed to implement the optuna distribution interface.
cv –
Cross-validation strategy. Possible inputs for cv are:
- integer to specify the number of folds in a CV splitter,
- a CV splitter,
- an iterable yielding (train, validation) splits as arrays of indices.
For integer, if estimator is a classifier and y is either binary or multiclass, sklearn.model_selection.StratifiedKFold is used. otherwise, sklearn.model_selection.KFold is used.
enable_pruning – If True, pruning is performed in the case where the underlying estimator supports partial_fit.
error_score – Value to assign to the score if an error occurs in fitting. If ‘raise’, the error is raised. If numeric, sklearn.exceptions.FitFailedWarning is raised. This does not affect the refit step, which will always raise the error.
max_iter – Maximum number of epochs. This is only used if the underlying estimator supports partial_fit.
n_jobs – Number of parallel jobs. -1 means using all processors.
n_trials – Number of trials. If None, there is no limitation on the number of trials. If timeout is also set to None, the study continues to create trials until it receives a termination signal such as Ctrl+C or SIGTERM. This trades off runtime vs quality of the solution.
random_state – Seed of the pseudo random number generator. If int, this is the seed used by the random number generator. If numpy.random.RandomState object, this is the random number generator. If None, the global random state from numpy.random is used.
refit – If True, refit the estimator with the best found hyperparameters. The refitted estimator is made available at the best_estimator_ attribute and permits using predict directly.
return_train_score – If True, training scores will be included. Computing training scores is used to get insights on how different hyperparameter settings impact the overfitting/underfitting trade-off. However computing training scores can be computationally expensive and is not strictly required to select the hyperparameters that yield the best generalization performance.
scoring – String or callable to evaluate the predictions on the validation data. If None, score on the estimator is used.
study – Study corresponds to the optimization task. If None, a new study is created.
subsample –
Proportion of samples that are used during hyperparameter search.
- If int, then draw subsample samples.
- If float, then draw subsample * X.shape[0] samples.
timeout – Time limit in seconds for the search of appropriate models. If None, the study is executed without time limitation. If n_trials is also set to None, the study continues to create trials until it receives a termination signal such as Ctrl+C or SIGTERM. This trades off runtime vs quality of the solution.
verbose – Verbosity level. The higher, the more messages.

best_estimator\_: Estimator that was chosen by the search. This is present only if refit is set to True.

n_splits\_: Number of cross-validation splits.

refit_time\_: Time for refitting the best estimator. This is present only if refit is set to True.

sample_indices\_: Indices of samples that are used during hyperparameter search.

scorer\_: Scorer function.

study\_: Actual study.

Examples

import optuna
from sklearn.datasets import load_iris
from sklearn.svm import SVC

clf = SVC(gamma='auto')
param_distributions = {
    'C': optuna.distributions.LogUniformDistribution(1e-10, 1e+10)
}
optuna_search = optuna.integration.OptunaSearchCV(
    clf,
    param_distributions
)
X, y = load_iris(return_X_y=True)
optuna_search.fit(X, y)
y_pred = optuna_search.predict(X)

property best_index_¶: Index which corresponds to the best candidate parameter setting.

property best_params_¶: Parameters of the best trial in the Study.

property best_score_¶: Mean cross-validated score of the best estimator.

property best_trial_¶: Best trial in the Study.

property classes_¶: Class labels.

property decision_function¶

Call decision_function on the best estimator.

This is available only if the underlying estimator supports decision_function and refit is set to True.

fit(X, y=None, groups=None, **fit_params)[source]¶

Run fit with all sets of parameters.

Parameters

X – Training data.
y – Target variable.
groups – Group labels for the samples used while splitting the dataset into train/validation set.
**fit_params – Parameters passed to fit on the estimator.

Returns

Return self.

Return type

self

property inverse_transform¶

Call inverse_transform on the best estimator.

This is available only if the underlying estimator supports inverse_transform and refit is set to True.

property n_trials_¶: Actual number of trials.

property predict¶

Call predict on the best estimator.

This is available only if the underlying estimator supports predict and refit is set to True.

property predict_log_proba¶

Call predict_log_proba on the best estimator.

This is available only if the underlying estimator supports predict_log_proba and refit is set to True.

property predict_proba¶

Call predict_proba on the best estimator.

This is available only if the underlying estimator supports predict_proba and refit is set to True.

score(X, y=None)[source]¶

Return the score on the given data.

Parameters

X – Data.
y – Target variable.

Returns

Scaler score.

Return type

score

property score_samples¶

Call score_samples on the best estimator.

This is available only if the underlying estimator supports score_samples and refit is set to True.

property set_user_attr¶: Call set_user_attr on the Study.

property transform¶

Call transform on the best estimator.

This is available only if the underlying estimator supports transform and refit is set to True.

property trials_¶: All trials in the Study.

property trials_dataframe¶: Call trials_dataframe on the Study.

property user_attrs_¶: User attributes in the Study.

class optuna.integration.AllenNLPExecutor(trial: optuna.trial._trial.Trial, config_file: str, serialization_dir: str, metrics: str = 'best_validation_accuracy', *, include_package: Union[str, List[str], None] = None)[source]¶

AllenNLP extension to use optuna with Jsonnet config file.

This feature is experimental since AllenNLP major release will come soon. The interface may change without prior notice to correspond to the update.

See the examples of objective function and config file.

Parameters

trial – A Trial corresponding to the current evaluation of the objective function.
config_file – Config file for AllenNLP. Hyperparameters should be masked with std.extVar. Please refer to the config example.
serialization_dir – A path which model weights and logs are saved.
metrics – An evaluation metric for the result of objective.
include_package – Additional packages to include. For more information, please see AllenNLP documentation.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

run() → float [source]¶: Train a model using AllenNLP.

optuna.integration.allennlp.dump_best_config(input_config_file: str, output_config_file: str, study: optuna.study.Study) → None [source]¶

Save JSON config file after updating with parameters from the best trial in the study.

Parameters

input_config_file – Input Jsonnet config file used with AllenNLPExecutor.
output_config_file – Output JSON config file.
study – Instance of Study. Note that optimize() must have been called.

optuna.logging¶

optuna.logging.get_verbosity() → int [source]¶

Return the current level for the Optuna’s root logger.

Returns: Logging level, e.g., optuna.logging.DEBUG and optuna.logging.INFO.

Note

Optuna has following logging levels:

optuna.logging.CRITICAL, optuna.logging.FATAL
optuna.logging.ERROR
optuna.logging.WARNING, optuna.logging.WARN
optuna.logging.INFO
optuna.logging.DEBUG

optuna.logging.set_verbosity(verbosity: int) → None [source]¶

Set the level for the Optuna’s root logger.

Parameters: verbosity – Logging level, e.g., optuna.logging.DEBUG and optuna.logging.INFO.

optuna.logging.disable_default_handler() → None [source]¶

Disable the default handler of the Optuna’s root logger.

Example

Stop and then resume logging to sys.stderr.

import optuna

study = optuna.create_study()

# There are no logs in sys.stderr.
optuna.logging.disable_default_handler()
study.optimize(objective, n_trials=10)

# There are logs in sys.stderr.
optuna.logging.enable_default_handler()
study.optimize(objective, n_trials=10)
# [I 2020-02-23 17:00:54,314] Trial 10 finished with value: ...
# [I 2020-02-23 17:00:54,356] Trial 11 finished with value: ...
# ...

optuna.logging.enable_default_handler() → None [source]¶

Enable the default handler of the Optuna’s root logger.

Please refer to the example shown in disable_default_handler().

optuna.logging.disable_propagation() → None [source]¶

Disable propagation of the library log outputs.

Note that log propagation is disabled by default.

optuna.logging.enable_propagation() → None [source]¶

Enable propagation of the library log outputs.

Please disable the Optuna’s default handler to prevent double logging if the root logger has been configured.

Example

Propagate all log output to the root logger in order to save them to the file.

import optuna
import logging

logger = logging.getLogger()

logger.setLevel(logging.INFO)  # Setup the root logger.
logger.addHandler(logging.FileHandler("foo.log", mode="w"))

optuna.logging.enable_propagation()  # Propagate logs to the root logger.
optuna.logging.disable_default_handler()  # Stop showing logs in sys.stderr.

study = optuna.create_study()

logger.info("Start optimization.")
study.optimize(objective, n_trials=10)

with open('foo.log') as f:
    assert f.readline() == "Start optimization.\n"
    assert f.readline().startswith("Trial 0 finished with value:")

optuna.multi_objective¶

optuna.multi_objective.samplers¶

class optuna.multi_objective.samplers.BaseMultiObjectiveSampler[source]¶

Base class for multi-objective samplers.

The abstract methods of this class are the same as ones defined by BaseSampler except for taking multi-objective versions of study and trial instances as the arguments.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

abstract infer_relative_search_space(study: optuna.multi_objective.study.MultiObjectiveStudy, trial: optuna.multi_objective.trial.FrozenMultiObjectiveTrial) → Dict[str, optuna.distributions.BaseDistribution][source]¶

Infer the search space that will be used by relative sampling in the target trial.

This method is called right before sample_relative() method, and the search space returned by this method is passed to it. The parameters not contained in the search space will be sampled by using sample_independent() method.

Parameters

study – Target study object.
trial – Target trial object.

Returns

A dictionary containing the parameter names and parameter’s distributions.

See also

Please refer to intersection_search_space() as an implementation of infer_relative_search_space().

abstract sample_independent(study: optuna.multi_objective.study.MultiObjectiveStudy, trial: optuna.multi_objective.trial.FrozenMultiObjectiveTrial, param_name: str, param_distribution: optuna.distributions.BaseDistribution) → Any[source]¶

Sample a parameter for a given distribution.

This method is called only for the parameters not contained in the search space returned by sample_relative() method. This method is suitable for sampling algorithms that do not use the relationship between parameters such as random sampling.

Parameters

study – Target study object.
trial – Target trial object.
param_name – Name of the sampled parameter.
param_distribution – Distribution object that specifies a prior and/or scale of the sampling algorithm.

Returns

A parameter value.

abstract sample_relative(study: optuna.multi_objective.study.MultiObjectiveStudy, trial: optuna.multi_objective.trial.FrozenMultiObjectiveTrial, search_space: Dict[str, optuna.distributions.BaseDistribution]) → Dict[str, Any][source]¶

Sample parameters in a given search space.

This method is called once at the beginning of each trial, i.e., right before the evaluation of the objective function. This method is suitable for sampling algorithms that use the relationship between parameters.

Parameters

study – Target study object.
trial – Target trial object.
search_space – The search space returned by infer_relative_search_space().

Returns

A dictionary containing the parameter names and the values.

class optuna.multi_objective.samplers.NSGAIIMultiObjectiveSampler(population_size: int = 50, mutation_prob: Optional[float] = None, crossover_prob: float = 0.9, swapping_prob: float = 0.5, seed: Optional[int] = None)[source]¶

Multi-objective sampler using the NSGA-II algorithm.

NSGA-II stands for “Nondominated Sorting Genetic Algorithm II”, which is a well known, fast and elitist multi-objective genetic algorithm.

For further information about NSGA-II, please refer to the following paper:

A fast and elitist multiobjective genetic algorithm: NSGA-II

Parameters

population_size – Number of individuals (trials) in a generation.
mutation_prob – Probability of mutating each parameter when creating a new individual. If None is specified, the value 1.0 / len(parent_trial.params) is used where parent_trial is the parent trial of the target individual.
crossover_prob – Probability that a crossover (parameters swapping between parents) will occur when creating a new individual.
swapping_prob – Probability of swapping each parameter of the parents during crossover.
seed – Seed for random number generator.

Note

Added in v1.5.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.5.0.

class optuna.multi_objective.samplers.RandomMultiObjectiveSampler(seed: Optional[int] = None)[source]¶

Multi-objective sampler using random sampling.

This sampler is based on independent sampling. See also BaseMultiObjectiveSampler for more details of ‘independent sampling’.

Example

import optuna
from optuna.multi_objective.samplers import RandomMultiObjectiveSampler

def objective(trial):
    x = trial.suggest_uniform('x', -5, 5)
    y = trial.suggest_uniform('y', -5, 5)
    return x ** 2, y + 10

study = optuna.multi_objective.create_study(
    ["minimize", "minimize"],
    sampler=RandomMultiObjectiveSampler()
)
study.optimize(objective, n_trials=10)

Args:: seed: Seed for random number generator.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

optuna.multi_objective.study¶

class optuna.multi_objective.study.MultiObjectiveStudy(study: optuna.study.Study)[source]¶

A study corresponds to a multi-objective optimization task, i.e., a set of trials.

This object provides interfaces to run a new Trial, access trials’ history, set/get user-defined attributes of the study itself.

Note that the direct use of this constructor is not recommended. To create and load a study, please refer to the documentation of create_study() and load_study() respectively.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

property directions¶

Return the optimization direction list.

Returns: A list that contains the optimization direction for each objective value.

enqueue_trial(params: Dict[str, Any]) → None [source]¶

Enqueue a trial with given parameter values.

You can fix the next sampling parameters which will be evaluated in your objective function.

Please refer to the documentation of optuna.study.Study.enqueue_trial() for further details.

Parameters: params – Parameter values to pass your objective function.

get_pareto_front_trials() → List[optuna.multi_objective.trial.FrozenMultiObjectiveTrial][source]¶

Return trials located at the pareto front in the study.

A trial is located at the pareto front if there are no trials that dominate the trial. It’s called that a trial t0 dominates another trial t1 if all(v0 <= v1) for v0, v1 in zip(t0.values, t1.values) and any(v0 < v1) for v0, v1 in zip(t0.values, t1.values) are held.

Returns: A list of FrozenMultiObjectiveTrial objects.

get_trials(deepcopy: bool = True) → List[optuna.multi_objective.trial.FrozenMultiObjectiveTrial][source]¶

Return all trials in the study.

The returned trials are ordered by trial number.

For library users, it’s recommended to use more handy trials property to get the trials instead.

Parameters: deepcopy – Flag to control whether to apply copy.deepcopy() to the trials. Note that if you set the flag to False, you shouldn’t mutate any fields of the returned trial. Otherwise the internal state of the study may corrupt and unexpected behavior may happen.
Returns: A list of FrozenMultiObjectiveTrial objects.

property n_objectives¶

Return the number of objectives.

Returns: Number of objectives.

optimize(objective: Callable[[multi_objective.trial.MultiObjectiveTrial], Sequence[float]], timeout: Optional[int] = None, n_trials: Optional[int] = None, n_jobs: int = 1, catch: Union[Tuple[()], Tuple[Type[Exception]]] = (), callbacks: Optional[List[Callable[[multi_objective.study.MultiObjectiveStudy, multi_objective.trial.FrozenMultiObjectiveTrial], None]]] = None, gc_after_trial: bool = True, show_progress_bar: bool = False) → None [source]¶

Optimize an objective function.

This method is the same as optuna.study.Study.optimize() except for taking an objective function that returns multi-objective values as the argument.

Please refer to the documentation of optuna.study.Study.optimize() for further details.

property sampler¶

Return the sampler.

Returns: A BaseMultiObjectiveSampler object.

set_user_attr(key: str, value: Any) → None [source]¶

Set a user attribute to the study.

Parameters

key – A key string of the attribute.
value – A value of the attribute. The value should be JSON serializable.

property trials¶

Return all trials in the study.

The returned trials are ordered by trial number.

This is a short form of self.get_trials(deepcopy=True).

Returns: A list of FrozenMultiObjectiveTrial objects.

property user_attrs¶

Return user attributes.

Returns: A dictionary containing all user attributes.

optuna.multi_objective.study.create_study(directions: List[str], study_name: Optional[str] = None, storage: Union[None, str, optuna.storages._base.BaseStorage] = None, sampler: Optional[multi_objective.samplers.BaseMultiObjectiveSampler] = None, load_if_exists: bool = False) → optuna.multi_objective.study.MultiObjectiveStudy [source]¶

Create a new MultiObjectiveStudy.

Parameters

directions – Optimization direction for each objective value. Set minimize for minimization and maximize for maximization.
study_name – Study’s name. If this argument is set to None, a unique name is generated automatically.
storage –
Database URL. If this argument is set to None, in-memory storage is used, and the Study will not be persistent.

Note

When a database URL is passed, Optuna internally uses SQLAlchemy to handle the database. Please refer to SQLAlchemy’s document for further details. If you want to specify non-default options to SQLAlchemy Engine, you can instantiate RDBStorage with your desired options and pass it to the storage argument instead of a URL.
sampler – A sampler object that implements background algorithm for value suggestion. If None is specified, NSGAIIMultiObjectiveSampler is used as the default. See also samplers.
load_if_exists – Flag to control the behavior to handle a conflict of study names. In the case where a study named study_name already exists in the storage, a DuplicatedStudyError is raised if load_if_exists is set to False. Otherwise, the creation of the study is skipped, and the existing one is returned.

Returns

A MultiObjectiveStudy object.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

optuna.multi_objective.study.load_study(study_name: str, storage: Union[str, optuna.storages._base.BaseStorage], sampler: Optional[multi_objective.samplers.BaseMultiObjectiveSampler] = None) → optuna.multi_objective.study.MultiObjectiveStudy [source]¶

Load the existing MultiObjectiveStudy that has the specified name.

Parameters

study_name – Study’s name. Each study has a unique name as an identifier.
storage – Database URL such as sqlite:///example.db. Please see also the documentation of create_study() for further details.
sampler – A sampler object that implements background algorithm for value suggestion. If None is specified, RandomMultiObjectiveSampler is used as the default. See also samplers.

Returns

A MultiObjectiveStudy object.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

optuna.multi_objective.trial¶

class optuna.multi_objective.trial.MultiObjectiveTrial(trial: optuna.trial._trial.Trial)[source]¶

A trial is a process of evaluating an objective function.

This object is passed to an objective function and provides interfaces to get parameter suggestion, manage the trial’s state, and set/get user-defined attributes of the trial.

Note that the direct use of this constructor is not recommended. This object is seamlessly instantiated and passed to the objective function behind the optuna.multi_objective.study.MultiObjectiveStudy.optimize() method; hence library users do not care about instantiation of this object.

Parameters: trial – A Trial object.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

property datetime_start¶

Return start datetime.

Returns: Datetime where the Trial started.

property distributions¶

Return distributions of parameters to be optimized.

Returns: A dictionary containing all distributions.

property number¶

Return trial’s number which is consecutive and unique in a study.

Returns: A trial number.

property params¶

Return parameters to be optimized.

Returns: A dictionary containing all parameters.

report(values: Sequence[float], step: int) → None [source]¶

Report intermediate objective function values for a given step.

The reported values are used by the pruners to determine whether this trial should be pruned.

See also

Please refer to BasePruner.

Note

The reported values are converted to float type by applying float() function internally. Thus, it accepts all float-like types (e.g., numpy.float32). If the conversion fails, a TypeError is raised.

Parameters

values – Intermediate objective function values for a given step.
step – Step of the trial (e.g., Epoch of neural network training).

set_user_attr(key: str, value: Any) → None [source]¶

Set user attributes to the trial.

Please refer to the documentation of optuna.trial.Trial.set_user_attr() for further details.

suggest_categorical(name: str, choices: Sequence[Union[None, bool, int, float, str]]) → Union[None, bool, int, float, str][source]¶

Suggest a value for the categorical parameter.

Please refer to the documentation of optuna.trial.Trial.suggest_categorical() for further details.

suggest_discrete_uniform(name: str, low: float, high: float, q: float) → float [source]¶

Suggest a value for the discrete parameter.

Please refer to the documentation of optuna.trial.Trial.suggest_discrete_uniform() for further details.

suggest_float(name: str, low: float, high: float, *, step: Optional[float] = None, log: bool = False) → float [source]¶

Suggest a value for the floating point parameter.

Please refer to the documentation of optuna.trial.Trial.suggest_float() for further details.

suggest_int(name: str, low: int, high: int, step: int = 1, log: bool = False) → int [source]¶

Suggest a value for the integer parameter.

Please refer to the documentation of optuna.trial.Trial.suggest_int() for further details.

suggest_loguniform(name: str, low: float, high: float) → float [source]¶

Suggest a value for the continuous parameter.

Please refer to the documentation of optuna.trial.Trial.suggest_loguniform() for further details.

suggest_uniform(name: str, low: float, high: float) → float [source]¶

Suggest a value for the continuous parameter.

Please refer to the documentation of optuna.trial.Trial.suggest_uniform() for further details.

property user_attrs¶

Return user attributes.

Returns: A dictionary containing all user attributes.

class optuna.multi_objective.trial.FrozenMultiObjectiveTrial(n_objectives: int, trial: optuna.trial._frozen.FrozenTrial)[source]¶

Status and results of a MultiObjectiveTrial.

number¶: Unique and consecutive number of MultiObjectiveTrial for each MultiObjectiveStudy. Note that this field uses zero-based numbering.

state¶: TrialState of the MultiObjectiveTrial.

values¶: Objective values of the MultiObjectiveTrial.

datetime_start¶: Datetime where the MultiObjectiveTrial started.

datetime_complete¶: Datetime where the MultiObjectiveTrial finished.

params¶: Dictionary that contains suggested parameters.

distributions¶: Dictionary that contains the distributions of params.

user_attrs¶: Dictionary that contains the attributes of the MultiObjectiveTrial set with optuna.multi_objective.trial.MultiObjectiveTrial.set_user_attr().

intermediate_values¶: Intermediate objective values set with optuna.multi_objective.trial.MultiObjectiveTrial.report().

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

optuna.multi_objective.visualization¶

Note

visualization module uses plotly to create figures, but JupyterLab cannot render them by default. Please follow this installation guide to show figures in JupyterLab.

optuna.multi_objective.visualization.plot_pareto_front(study: optuna.multi_objective.study.MultiObjectiveStudy, names: Optional[List[str]] = None) → go.Figure[source]¶

Plot the pareto front of a study.

Example

The following code snippet shows how to plot the pareto front of a study.

import optuna

def objective(trial):
   x = trial.suggest_float("x", 0, 5)
   y = trial.suggest_float("y", 0, 3)

   v0 = (4 * x) ** 2 + (4 * y) ** 2
   v1 = (x - 5) ** 2 + (y - 5) ** 2
   return v0, v1

study = optuna.multi_objective.create_study(["minimize", "minimize"])
study.optimize(objective, n_trials=50)

optuna.multi_objective.visualization.plot_pareto_front(study)

Parameters

study – A MultiObjectiveStudy object whose trials are plotted for their objective values.
names – Objective name list used as the axis titles. If None is specified, “Objective {objective_index}” is used instead.

Returns

A plotly.graph_objs.Figure object.

Raises

ValueError – If the number of objectives of study isn’t 2 or 3.

Note

Added in v2.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.0.0.

optuna.pruners¶

class optuna.pruners.BasePruner[source]¶

Base class for pruners.

abstract prune(study, trial)[source]¶

Judge whether the trial should be pruned based on the reported values.

Note that this method is not supposed to be called by library users. Instead, optuna.trial.Trial.report() and optuna.trial.Trial.should_prune() provide user interfaces to implement pruning mechanism in an objective function.

Parameters

study – Study object of the target study.
trial – FrozenTrial object of the target trial. Take a copy before modifying this object.

Returns

A boolean value representing whether the trial should be pruned.

class optuna.pruners.MedianPruner(n_startup_trials=5, n_warmup_steps=0, interval_steps=1)[source]¶

Pruner using the median stopping rule.

Prune if the trial’s best intermediate result is worse than median of intermediate results of previous trials at the same step.

Example

We minimize an objective function with the median stopping rule.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)
classes = np.unique(y)

def objective(trial):
    alpha = trial.suggest_uniform('alpha', 0.0, 1.0)
    clf = SGDClassifier(alpha=alpha)
    n_train_iter = 100

    for step in range(n_train_iter):
        clf.partial_fit(X_train, y_train, classes=classes)

        intermediate_value = clf.score(X_valid, y_valid)
        trial.report(intermediate_value, step)

        if trial.should_prune():
            raise optuna.TrialPruned()

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize',
                            pruner=optuna.pruners.MedianPruner(n_startup_trials=5,
                                                               n_warmup_steps=30,
                                                               interval_steps=10))
study.optimize(objective, n_trials=20)

Parameters

n_startup_trials – Pruning is disabled until the given number of trials finish in the same study.
n_warmup_steps – Pruning is disabled until the trial exceeds the given number of step.
interval_steps – Interval in number of steps between the pruning checks, offset by the warmup steps. If no value has been reported at the time of a pruning check, that particular check will be postponed until a value is reported.

class optuna.pruners.NopPruner[source]¶

Pruner which never prunes trials.

Example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)
classes = np.unique(y)

def objective(trial):
    alpha = trial.suggest_uniform('alpha', 0.0, 1.0)
    clf = SGDClassifier(alpha=alpha)
    n_train_iter = 100

    for step in range(n_train_iter):
        clf.partial_fit(X_train, y_train, classes=classes)

        intermediate_value = clf.score(X_valid, y_valid)
        trial.report(intermediate_value, step)

        if trial.should_prune():
            assert False, "should_prune() should always return False with this pruner."
            raise optuna.TrialPruned()

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize',
                            pruner=optuna.pruners.NopPruner())
study.optimize(objective, n_trials=20)

class optuna.pruners.PercentilePruner(percentile, n_startup_trials=5, n_warmup_steps=0, interval_steps=1)[source]¶

Pruner to keep the specified percentile of the trials.

Prune if the best intermediate value is in the bottom percentile among trials at the same step.

Example

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)
classes = np.unique(y)

def objective(trial):
    alpha = trial.suggest_uniform('alpha', 0.0, 1.0)
    clf = SGDClassifier(alpha=alpha)
    n_train_iter = 100

    for step in range(n_train_iter):
        clf.partial_fit(X_train, y_train, classes=classes)

        intermediate_value = clf.score(X_valid, y_valid)
        trial.report(intermediate_value, step)

        if trial.should_prune():
            raise optuna.TrialPruned()

    return clf.score(X_valid, y_valid)

study = optuna.create_study(
    direction='maximize',
    pruner=optuna.pruners.PercentilePruner(25.0, n_startup_trials=5,
                                           n_warmup_steps=30, interval_steps=10))
study.optimize(objective, n_trials=20)

Parameters

percentile – Percentile which must be between 0 and 100 inclusive (e.g., When given 25.0, top of 25th percentile trials are kept).
n_startup_trials – Pruning is disabled until the given number of trials finish in the same study.
n_warmup_steps – Pruning is disabled until the trial exceeds the given number of step.
interval_steps – Interval in number of steps between the pruning checks, offset by the warmup steps. If no value has been reported at the time of a pruning check, that particular check will be postponed until a value is reported. Value must be at least 1.

class optuna.pruners.SuccessiveHalvingPruner(min_resource='auto', reduction_factor=4, min_early_stopping_rate=0)[source]¶

Pruner using Asynchronous Successive Halving Algorithm.

Successive Halving is a bandit-based algorithm to identify the best one among multiple configurations. This class implements an asynchronous version of Successive Halving. Please refer to the paper of Asynchronous Successive Halving for detailed descriptions.

Note that, this class does not take care of the parameter for the maximum resource, referred to as $R$ in the paper. The maximum resource allocated to a trial is typically limited inside the objective function (e.g., step number in simple.py, EPOCH number in chainer_integration.py).

Example

We minimize an objective function with SuccessiveHalvingPruner.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)
classes = np.unique(y)

def objective(trial):
    alpha = trial.suggest_uniform('alpha', 0.0, 1.0)
    clf = SGDClassifier(alpha=alpha)
    n_train_iter = 100

    for step in range(n_train_iter):
        clf.partial_fit(X_train, y_train, classes=classes)

        intermediate_value = clf.score(X_valid, y_valid)
        trial.report(intermediate_value, step)

        if trial.should_prune():
            raise optuna.TrialPruned()

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize',
                            pruner=optuna.pruners.SuccessiveHalvingPruner())
study.optimize(objective, n_trials=20)

Parameters

min_resource –
A parameter for specifying the minimum resource allocated to a trial (in the paper this parameter is referred to as $r$). This parameter defaults to ‘auto’ where the value is determined based on a heuristic that looks at the number of required steps for the first trial to complete.

A trial is never pruned until it executes $\mathsf{min}\_\mathsf{resource} \times \mathsf{reduction}\_\mathsf{factor}^{ \mathsf{min}\_\mathsf{early}\_\mathsf{stopping}\_\mathsf{rate}}$ steps (i.e., the completion point of the first rung). When the trial completes the first rung, it will be promoted to the next rung only if the value of the trial is placed in the top ${1 \over \mathsf{reduction}\_\mathsf{factor}}$ fraction of the all trials that already have reached the point (otherwise it will be pruned there). If the trial won the competition, it runs until the next completion point (i.e., $\mathsf{min}\_\mathsf{resource} \times \mathsf{reduction}\_\mathsf{factor}^{ (\mathsf{min}\_\mathsf{early}\_\mathsf{stopping}\_\mathsf{rate} + \mathsf{rung})}$ steps) and repeats the same procedure.

Note

If the step of the last intermediate value may change with each trial, please manually specify the minimum possible step to min_resource.
reduction_factor –
A parameter for specifying reduction factor of promotable trials (in the paper this parameter is referred to as $\eta$). At the completion point of each rung, about ${1 \over \mathsf{reduction}\_\mathsf{factor}}$ trials will be promoted.
min_early_stopping_rate –
A parameter for specifying the minimum early-stopping rate (in the paper this parameter is referred to as $s$).

class optuna.pruners.HyperbandPruner(min_resource: int = 1, max_resource: Union[str, int] = 'auto', reduction_factor: int = 3)[source]¶

Pruner using Hyperband.

As SuccessiveHalving (SHA) requires the number of configurations $n$ as its hyperparameter. For a given finite budget $B$, all the configurations have the resources of $B \over n$ on average. As you can see, there will be a trade-off of $B$ and $B \over n$. Hyperband attacks this trade-off by trying different $n$ values for a fixed budget.

Note

In the Hyperband paper, the counterpart of RandomSampler is used.
Optuna uses TPESampler by default.
The benchmark result shows that optuna.pruners.HyperbandPruner supports both samplers.

Note

If you use HyperbandPruner with TPESampler, it’s recommended to consider to set larger n_trials or timeout to make full use of the characteristics of TPESampler because TPESampler uses some (by default, $10$) Trials for its startup.

As Hyperband runs multiple SuccessiveHalvingPruner and collect trials based on the current Trial’s bracket ID, each bracket needs to observe more than $10$ Trials for TPESampler to adapt its search space.

Thus, for example, if HyperbandPruner has $4$ pruners in it, at least $4 \times 10$ trials are consumed for startup.

Note

Hyperband has several SuccessiveHalvingPruner. Each SuccessiveHalvingPruner is referred as “bracket” in the original paper. The number of brackets is an important factor to control the early stopping behavior of Hyperband and is automatically determined by min_resource, max_resource and reduction_factor as The number of brackets = floor(log_{reduction_factor}(max_resource / min_resource)) + 1. Please set reduction_factor so that the number of brackets is not too large　(about 4 ~ 6 in most use cases).　Please see Section 3.6 of the original paper for the detail.

Example

We minimize an objective function with Hyperband pruning algorithm.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y)
classes = np.unique(y)
n_train_iter = 100

def objective(trial):
    alpha = trial.suggest_uniform('alpha', 0.0, 1.0)
    clf = SGDClassifier(alpha=alpha)

    for step in range(n_train_iter):
        clf.partial_fit(X_train, y_train, classes=classes)

        intermediate_value = clf.score(X_valid, y_valid)
        trial.report(intermediate_value, step)

        if trial.should_prune():
            raise optuna.TrialPruned()

    return clf.score(X_valid, y_valid)

study = optuna.create_study(
    direction='maximize',
    pruner=optuna.pruners.HyperbandPruner(
        min_resource=1,
        max_resource=n_train_iter,
        reduction_factor=3
    )
)
study.optimize(objective, n_trials=20)

Parameters

min_resource – A parameter for specifying the minimum resource allocated to a trial noted as $r$ in the paper. A smaller $r$ will give a result faster, but a larger $r$ will give a better guarantee of successful judging between configurations. See the details for SuccessiveHalvingPruner.
max_resource –
A parameter for specifying the maximum resource allocated to a trial. $R$ in the paper corresponds to max_resource / min_resource. This value represents and should match the maximum iteration steps (e.g., the number of epochs for neural networks). When this argument is “auto”, the maximum resource is estimated according to the completed trials. The default value of this argument is “auto”.

Note

With “auto”, the maximum resource will be the largest step reported by report() in the first, or one of the first if trained in parallel, completed trial. No trials will be pruned until the maximum resource is determined.

Note

If the step of the last intermediate value may change with each trial, please manually specify the maximum possible step to max_resource.
reduction_factor – A parameter for specifying reduction factor of promotable trials noted as $\eta$ in the paper. See the details for SuccessiveHalvingPruner.

Note

Added in v1.1.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.1.0.

class optuna.pruners.ThresholdPruner(lower: Optional[float] = None, upper: Optional[float] = None, n_warmup_steps: int = 0, interval_steps: int = 1)[source]¶

Pruner to detect outlying metrics of the trials.

Prune if a metric exceeds upper threshold, falls behind lower threshold or reaches nan.

Example

from optuna import create_study
from optuna.pruners import ThresholdPruner
from optuna import TrialPruned

def objective_for_upper(trial):
    for step, y in enumerate(ys_for_upper):
        trial.report(y, step)

        if trial.should_prune():
            raise TrialPruned()
    return ys_for_upper[-1]


def objective_for_lower(trial):
    for step, y in enumerate(ys_for_lower):
        trial.report(y, step)

        if trial.should_prune():
            raise TrialPruned()
    return ys_for_lower[-1]


ys_for_upper = [0.0, 0.1, 0.2, 0.5, 1.2]
ys_for_lower = [100.0, 90.0, 0.1, 0.0, -1]
n_trial_step = 5

study = create_study(pruner=ThresholdPruner(upper=1.0))
study.optimize(objective_for_upper, n_trials=10)

study = create_study(pruner=ThresholdPruner(lower=0.0))
study.optimize(objective_for_lower, n_trials=10)

Args

lower:: A minimum value which determines whether pruner prunes or not. If an intermediate value is smaller than lower, it prunes.
upper:: A maximum value which determines whether pruner prunes or not. If an intermediate value is larger than upper, it prunes.
n_warmup_steps:: Pruning is disabled until the trial exceeds the given number of step.
interval_steps:: Interval in number of steps between the pruning checks, offset by the warmup steps. If no value has been reported at the time of a pruning check, that particular check will be postponed until a value is reported. Value must be at least 1.

optuna.samplers¶

class optuna.samplers.BaseSampler[source]¶

Base class for samplers.

Optuna combines two types of sampling strategies, which are called relative sampling and independent sampling.

The relative sampling determines values of multiple parameters simultaneously so that sampling algorithms can use relationship between parameters (e.g., correlation). Target parameters of the relative sampling are described in a relative search space, which is determined by infer_relative_search_space().

The independent sampling determines a value of a single parameter without considering any relationship between parameters. Target parameters of the independent sampling are the parameters not described in the relative search space.

More specifically, parameters are sampled by the following procedure. At the beginning of a trial, infer_relative_search_space() is called to determine the relative search space for the trial. Then, sample_relative() is invoked to sample parameters from the relative search space. During the execution of the objective function, sample_independent() is used to sample parameters that don’t belong to the relative search space.

The following figure depicts the lifetime of a trial and how the above three methods are called in the trial.

abstract infer_relative_search_space(study, trial)[source]¶

Infer the search space that will be used by relative sampling in the target trial.

This method is called right before sample_relative() method, and the search space returned by this method is pass to it. The parameters not contained in the search space will be sampled by using sample_independent() method.

Parameters

study – Target study object.
trial – Target trial object. Take a copy before modifying this object.

Returns

A dictionary containing the parameter names and parameter’s distributions.

See also

Please refer to intersection_search_space() as an implementation of infer_relative_search_space().

reseed_rng() → None [source]¶

Reseed sampler’s random number generator.

This method is called by the Study instance if trials are executed in parallel with the option n_jobs>1. In that case, the sampler instance will be replicated including the state of the random number generator, and they may suggest the same values. To prevent this issue, this method assigns a different seed to each random number generator.

abstract sample_independent(study, trial, param_name, param_distribution)[source]¶

Sample a parameter for a given distribution.

This method is called only for the parameters not contained in the search space returned by sample_relative() method. This method is suitable for sampling algorithms that do not use relationship between parameters such as random sampling and TPE.

Note

The failed trials are ignored by any build-in samplers when they sample new parameters. Thus, failed trials are regarded as deleted in the samplers’ perspective.

Parameters

study – Target study object.
trial – Target trial object. Take a copy before modifying this object.
param_name – Name of the sampled parameter.
param_distribution – Distribution object that specifies a prior and/or scale of the sampling algorithm.

Returns

A parameter value.

abstract sample_relative(study, trial, search_space)[source]¶

Sample parameters in a given search space.

This method is called once at the beginning of each trial, i.e., right before the evaluation of the objective function. This method is suitable for sampling algorithms that use relationship between parameters such as Gaussian Process and CMA-ES.

Note

The failed trials are ignored by any build-in samplers when they sample new parameters. Thus, failed trials are regarded as deleted in the samplers’ perspective.

Parameters

study – Target study object.
trial – Target trial object. Take a copy before modifying this object.
search_space – The search space returned by infer_relative_search_space().

Returns

A dictionary containing the parameter names and the values.

class optuna.samplers.GridSampler(search_space)[source]¶

Sampler using grid search.

With GridSampler, the trials suggest all combinations of parameters in the given search space during the study.

Example

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -100, 100)
    y = trial.suggest_int('y', -100, 100)
    return x ** 2 + y ** 2

search_space = {
    'x': [-50, 0, 50],
    'y': [-99, 0, 99]
}
study = optuna.create_study(sampler=optuna.samplers.GridSampler(search_space))
study.optimize(objective, n_trials=3*3)

Note

GridSampler raises an error if all combinations in the passed search_space has already been evaluated. Please make sure that unnecessary trials do not run during optimization by properly setting n_trials in the optimize() method.

Note

GridSampler does not take care of a parameter’s quantization specified by discrete suggest methods but just samples one of values specified in the search space. E.g., in the following code snippet, either of -0.5 or 0.5 is sampled as x instead of an integer point.

import optuna

def objective(trial):
    # The following suggest method specifies integer points between -5 and 5.
    x = trial.suggest_discrete_uniform('x', -5, 5, 1)
    return x ** 2

# Non-int points are specified in the grid.
search_space = {'x': [-0.5, 0.5]}
study = optuna.create_study(sampler=optuna.samplers.GridSampler(search_space))
study.optimize(objective, n_trials=2)

Parameters: search_space – A dictionary whose key and value are a parameter name and the corresponding candidates of values, respectively.

Note

Added in v1.2.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.2.0.

class optuna.samplers.RandomSampler(seed=None)[source]¶

Sampler using random sampling.

This sampler is based on independent sampling. See also BaseSampler for more details of ‘independent sampling’.

Example

import optuna
from optuna.samplers import RandomSampler

def objective(trial):
    x = trial.suggest_uniform('x', -5, 5)
    return x**2

study = optuna.create_study(sampler=RandomSampler())
study.optimize(objective, n_trials=10)

Args:: seed: Seed for random number generator.

reseed_rng() → None [source]¶

Reseed sampler’s random number generator.

This method is called by the Study instance if trials are executed in parallel with the option n_jobs>1. In that case, the sampler instance will be replicated including the state of the random number generator, and they may suggest the same values. To prevent this issue, this method assigns a different seed to each random number generator.

class optuna.samplers.TPESampler(consider_prior=True, prior_weight=1.0, consider_magic_clip=True, consider_endpoints=False, n_startup_trials=10, n_ei_candidates=24, gamma=<function default_gamma>, weights=<function default_weights>, seed=None)[source]¶

Sampler using TPE (Tree-structured Parzen Estimator) algorithm.

This sampler is based on independent sampling. See also BaseSampler for more details of ‘independent sampling’.

On each trial, for each parameter, TPE fits one Gaussian Mixture Model (GMM) l(x) to the set of parameter values associated with the best objective values, and another GMM g(x) to the remaining parameter values. It chooses the parameter value x that maximizes the ratio l(x)/g(x).

For further information about TPE algorithm, please refer to the following papers:

Example

import optuna
from optuna.samplers import TPESampler

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return x**2

study = optuna.create_study(sampler=TPESampler())
study.optimize(objective, n_trials=10)

Parameters

consider_prior – Enhance the stability of Parzen estimator by imposing a Gaussian prior when True. The prior is only effective if the sampling distribution is either UniformDistribution, DiscreteUniformDistribution, LogUniformDistribution, IntUniformDistribution, or IntLogUniformDistribution.
prior_weight – The weight of the prior. This argument is used in UniformDistribution, DiscreteUniformDistribution, LogUniformDistribution, IntUniformDistribution, IntLogUniformDistribution, and CategoricalDistribution.
consider_magic_clip – Enable a heuristic to limit the smallest variances of Gaussians used in the Parzen estimator.
consider_endpoints – Take endpoints of domains into account when calculating variances of Gaussians in Parzen estimator. See the original paper for details on the heuristics to calculate the variances.
n_startup_trials – The random sampling is used instead of the TPE algorithm until the given number of trials finish in the same study.
n_ei_candidates – Number of candidate samples used to calculate the expected improvement.
gamma – A function that takes the number of finished trials and returns the number of trials to form a density function for samples with low grains. See the original paper for more details.
weights –
A function that takes the number of finished trials and returns a weight for them. See Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures for more details.
seed – Seed for random number generator.

static hyperopt_parameters()[source]¶

Return the the default parameters of hyperopt (v0.1.2).

TPESampler can be instantiated with the parameters returned by this method.

Example

Create a TPESampler instance with the default parameters of hyperopt.

import optuna
from optuna.samplers import TPESampler

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return x**2

sampler = TPESampler(**TPESampler.hyperopt_parameters())
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=10)

Returns: A dictionary containing the default parameters of hyperopt.

reseed_rng() → None [source]¶

Reseed sampler’s random number generator.

This method is called by the Study instance if trials are executed in parallel with the option n_jobs>1. In that case, the sampler instance will be replicated including the state of the random number generator, and they may suggest the same values. To prevent this issue, this method assigns a different seed to each random number generator.

class optuna.samplers.CmaEsSampler(x0: Optional[Dict[str, Any]] = None, sigma0: Optional[float] = None, n_startup_trials: int = 1, independent_sampler: Optional[optuna.samplers._base.BaseSampler] = None, warn_independent_sampling: bool = True, seed: Optional[int] = None, *, consider_pruned_trials: bool = False)[source]¶

A Sampler using CMA-ES algorithm.

Example

Optimize a simple quadratic function by using CmaEsSampler.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -1, 1)
    y = trial.suggest_int('y', -1, 1)
    return x ** 2 + y

sampler = optuna.samplers.CmaEsSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=20)

Please note that this sampler does not support CategoricalDistribution. If your search space contains categorical parameters, I recommend you to use TPESampler instead. Furthermore, there is room for performance improvements in parallel optimization settings. This sampler cannot use some trials for updating the parameters of multivariate normal distribution.

See also

You can also use optuna.integration.CmaEsSampler which is a sampler using cma library as the backend.

Parameters

x0 – A dictionary of an initial parameter values for CMA-ES. By default, the mean of low and high for each distribution is used.
sigma0 – Initial standard deviation of CMA-ES. By default, sigma0 is set to min_range / 6, where min_range denotes the minimum range of the distributions in the search space.
seed – A random seed for CMA-ES.
n_startup_trials – The independent sampling is used instead of the CMA-ES algorithm until the given number of trials finish in the same study.
independent_sampler –
A BaseSampler instance that is used for independent sampling. The parameters not contained in the relative search space are sampled by this sampler. The search space for CmaEsSampler is determined by intersection_search_space().

If None is specified, RandomSampler is used as the default.

See also

optuna.samplers module provides built-in independent samplers such as RandomSampler and TPESampler.
warn_independent_sampling –
If this is True, a warning message is emitted when the value of a parameter is sampled by using an independent sampler.

Note that the parameters of the first trial in a study are always sampled via an independent sampler, so no warning messages are emitted in this case.
consider_pruned_trials –
If this is True, the PRUNED trials are considered for sampling.

Note

Added in v2.0.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v2.0.0.

Note

It is suggested to set this flag False when the MedianPruner is used. On the other hand, it is suggested to set this flag True when the HyperbandPruner is used. Please see the benchmark result for the details.

reseed_rng() → None [source]¶

Reseed sampler’s random number generator.

This method is called by the Study instance if trials are executed in parallel with the option n_jobs>1. In that case, the sampler instance will be replicated including the state of the random number generator, and they may suggest the same values. To prevent this issue, this method assigns a different seed to each random number generator.

class optuna.samplers.IntersectionSearchSpace[source]¶

A class to calculate the intersection search space of a BaseStudy.

Intersection search space contains the intersection of parameter distributions that have been suggested in the completed trials of the study so far. If there are multiple parameters that have the same name but different distributions, neither is included in the resulting search space (i.e., the parameters with dynamic value ranges are excluded).

Note that an instance of this class is supposed to be used for only one study. If different studies are passed to calculate(), a ValueError is raised.

calculate(study: optuna.study.BaseStudy, ordered_dict: bool = False) → Dict[str, optuna.distributions.BaseDistribution][source]¶

Returns the intersection search space of the BaseStudy.

Parameters

study – A study with completed trials.
ordered_dict – A boolean flag determining the return type. If False, the returned object will be a dict. If True, the returned object will be an collections.OrderedDict sorted by keys, i.e. parameter names.

Returns

A dictionary containing the parameter names and parameter’s distributions.

optuna.samplers.intersection_search_space(study: optuna.study.BaseStudy, ordered_dict: bool = False) → Dict[str, optuna.distributions.BaseDistribution][source]¶

Return the intersection search space of the BaseStudy.

Intersection search space contains the intersection of parameter distributions that have been suggested in the completed trials of the study so far. If there are multiple parameters that have the same name but different distributions, neither is included in the resulting search space (i.e., the parameters with dynamic value ranges are excluded).

Note

IntersectionSearchSpace provides the same functionality with a much faster way. Please consider using it if you want to reduce execution time as much as possible.

Parameters

study – A study with completed trials.
ordered_dict – A boolean flag determining the return type. If False, the returned object will be a dict. If True, the returned object will be an collections.OrderedDict sorted by keys, i.e. parameter names.

Returns

A dictionary containing the parameter names and parameter’s distributions.

optuna.storages¶

class optuna.storages.RDBStorage(url, engine_kwargs=None, skip_compatibility_check=False)[source]¶

Storage class for RDB backend.

Note that library users can instantiate this class, but the attributes provided by this class are not supposed to be directly accessed by them.

Example

Create an RDBStorage instance with customized pool_size and timeout settings.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -100, 100)
    return x ** 2

storage = optuna.storages.RDBStorage(
    url='sqlite:///:memory:',
    engine_kwargs={
        'pool_size': 20,
        'connect_args': {
            'timeout': 10
        }
    }
)

study = optuna.create_study(storage=storage)
study.optimize(objective, n_trials=10)

Parameters

url – URL of the storage.
engine_kwargs – A dictionary of keyword arguments that is passed to sqlalchemy.engine.create_engine function.
skip_compatibility_check – Flag to skip schema compatibility check if set to True.

Note

If you use MySQL, pool_pre_ping will be set to True by default to prevent connection timeout. You can turn it off with engine_kwargs['pool_pre_ping']=False, but it is recommended to keep the setting if execution time of your objective function is longer than the wait_timeout of your MySQL configuration.

class optuna.storages.RedisStorage(url)[source]¶

Storage class for Redis backend.

Note that library users can instantiate this class, but the attributes provided by this class are not supposed to be directly accessed by them.

Example

We create an RedisStorage instance using the given redis database URL.

>>> import optuna
>>>
>>> def objective(trial):
>>>     ...
>>>
>>> storage = optuna.storages.RedisStorage(
>>>     url='redis://passwd@localhost:port/db',
>>> )
>>>
>>> study = optuna.create_study(storage=storage)
>>> study.optimize(objective)

Parameters: url – URL of the redis storage, password and db are optional. (ie: redis://localhost:6379)

Note

If you use plan to use Redis as a storage mechanism for optuna, make sure Redis in installed and running. Please execute $ pip install -U redis to install redis python library.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

optuna.structs¶

class optuna.structs.TrialState[source]¶

State of a Trial.

RUNNING¶: The Trial is running.

COMPLETE¶: The Trial has been finished without any error.

PRUNED¶: The Trial has been pruned with TrialPruned.

FAIL¶: The Trial has failed due to an uncaught error.

Deprecated since version 1.4.0: This class is deprecated. Please use TrialState instead.

class optuna.structs.StudyDirection[source]¶

Direction of a Study.

NOT_SET¶: Direction has not been set.

MINIMIZE¶: Study minimizes the objective function.

MAXIMIZE¶: Study maximizes the objective function.

Deprecated since version 1.4.0: This class is deprecated. Please use StudyDirection instead.

class optuna.structs.FrozenTrial(number, state, value, datetime_start, datetime_complete, params, distributions, user_attrs, system_attrs, intermediate_values, trial_id)[source]¶

Status and results of a Trial.

Deprecated since version 1.4.0: This class was moved to trial. Please use FrozenTrial instead.

number¶: Unique and consecutive number of Trial for each Study. Note that this field uses zero-based numbering.

state¶: TrialState of the Trial.

value¶: Objective value of the Trial.

datetime_start¶: Datetime where the Trial started.

datetime_complete¶: Datetime where the Trial finished.

params¶: Dictionary that contains suggested parameters.

user_attrs¶: Dictionary that contains the attributes of the Trial set with optuna.trial.Trial.set_user_attr().

intermediate_values¶: Intermediate objective values set with optuna.trial.Trial.report().

property distributions¶: Dictionary that contains the distributions of params.

property duration¶

Return the elapsed time taken to complete the trial.

Returns: The duration.

class optuna.structs.StudySummary(study_name, direction, best_trial, user_attrs, system_attrs, n_trials, datetime_start, study_id)[source]¶

Basic attributes and aggregated results of a Study.

Deprecated since version 1.4.0: This class was moved to study. Please use StudySummary instead.

study_name¶: Name of the Study.

direction¶: StudyDirection of the Study.

best_trial¶: FrozenTrial with best objective value in the Study.

user_attrs¶: Dictionary that contains the attributes of the Study set with optuna.study.Study.set_user_attr().

system_attrs¶: Dictionary that contains the attributes of the Study internally set by Optuna.

n_trials¶: The number of trials ran in the Study.

datetime_start¶: Datetime where the Study started.

optuna.study¶

class optuna.study.Study(study_name, storage, sampler=None, pruner=None)[source]¶

A study corresponds to an optimization task, i.e., a set of trials.

This object provides interfaces to run a new Trial, access trials’ history, set/get user-defined attributes of the study itself.

Note that the direct use of this constructor is not recommended. To create and load a study, please refer to the documentation of create_study() and load_study() respectively.

property best_params¶

Return parameters of the best trial in the study.

Returns: A dictionary containing parameters of the best trial.

property best_trial¶

Return the best trial in the study.

Returns: A FrozenTrial object of the best trial.

property best_value¶

Return the best objective value in the study.

Returns: A float representing the best objective value.

property direction¶

Return the direction of the study.

Returns: A StudyDirection object.

enqueue_trial(params)[source]¶

Enqueue a trial with given parameter values.

You can fix the next sampling parameters which will be evaluated in your objective function.

Example

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', 0, 10)
    return x ** 2

study = optuna.create_study()
study.enqueue_trial({'x': 5})
study.enqueue_trial({'x': 0})
study.optimize(objective, n_trials=2)

assert study.trials[0].params == {'x': 5}
assert study.trials[1].params == {'x': 0}

Parameters: params – Parameter values to pass your objective function.

Note

Added in v1.2.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.2.0.

get_trials(deepcopy=True)¶

Return all trials in the study.

The returned trials are ordered by trial number.

For library users, it’s recommended to use more handy trials property to get the trials instead.

Parameters: deepcopy – Flag to control whether to apply copy.deepcopy() to the trials. Note that if you set the flag to False, you shouldn’t mutate any fields of the returned trial. Otherwise the internal state of the study may corrupt and unexpected behavior may happen.
Returns: A list of FrozenTrial objects.

optimize(func, n_trials=None, timeout=None, n_jobs=1, catch=(), callbacks=None, gc_after_trial=False, show_progress_bar=False)[source]¶

Optimize an objective function.

Optimization is done by choosing a suitable set of hyperparameter values from a given range. Uses a sampler which implements the task of value suggestion based on a specified distribution. The sampler is specified in create_study() and the default choice for the sampler is TPE. See also TPESampler for more details on ‘TPE’.

Parameters

func – A callable that implements objective function.
n_trials – The number of trials. If this argument is set to None, there is no limitation on the number of trials. If timeout is also set to None, the study continues to create trials until it receives a termination signal such as Ctrl+C or SIGTERM.
timeout – Stop study after the given number of second(s). If this argument is set to None, the study is executed without time limitation. If n_trials is also set to None, the study continues to create trials until it receives a termination signal such as Ctrl+C or SIGTERM.
n_jobs – The number of parallel jobs. If this argument is set to -1, the number is set to CPU count.
catch – A study continues to run even when a trial raises one of the exceptions specified in this argument. Default is an empty tuple, i.e. the study will stop for any exception except for TrialPruned.
callbacks – List of callback functions that are invoked at the end of each trial. Each function must accept two parameters with the following types in this order: Study and FrozenTrial.
gc_after_trial –
Flag to determine whether to automatically run garbage collection after each trial. Set to True to run the garbage collection, False otherwise. When it runs, it runs a full collection by internally calling gc.collect(). If you see an increase in memory consumption over several trials, try setting this flag to True.

See also

How do I avoid running out of memory (OOM) when optimizing studies?
show_progress_bar – Flag to show progress bars or not. To disable progress bar, set this False. Currently, progress bar is experimental feature and disabled when n_jobs $\ne 1$.

set_user_attr(key, value)[source]¶

Set a user attribute to the study.

Parameters

key – A key string of the attribute.
value – A value of the attribute. The value should be JSON serializable.

stop() → None [source]¶

Exit from the current optimization loop after the running trials finish.

This method lets the running optimize() method return immediately after all trials which the optimize() method spawned finishes. This method does not affect any behaviors of parallel or successive study processes.

Raises: RuntimeError – If this method is called outside an objective function or callback.

Note

Added in v1.4.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

property trials¶

Return all trials in the study.

The returned trials are ordered by trial number.

This is a short form of self.get_trials(deepcopy=True).

Returns: A list of FrozenTrial objects.

trials_dataframe(attrs='number', 'value', 'datetime_start', 'datetime_complete', 'duration', 'params', 'user_attrs', 'system_attrs', 'state', multi_index=False)[source]¶

Export trials as a pandas DataFrame.

The DataFrame provides various features to analyze studies. It is also useful to draw a histogram of objective values and to export trials as a CSV file. If there are no trials, an empty DataFrame is returned.

Example

import optuna
import pandas

def objective(trial):
    x = trial.suggest_uniform('x', -1, 1)
    return x ** 2

study = optuna.create_study()
study.optimize(objective, n_trials=3)

# Create a dataframe from the study.
df = study.trials_dataframe()
assert isinstance(df, pandas.DataFrame)
assert df.shape[0] == 3  # n_trials.

Parameters

attrs – Specifies field names of FrozenTrial to include them to a DataFrame of trials.
multi_index – Specifies whether the returned DataFrame employs MultiIndex or not. Columns that are hierarchical by nature such as (params, x) will be flattened to params_x when set to False.

Returns

A pandas DataFrame of trials in the Study.

property user_attrs¶

Return user attributes.

Returns: A dictionary containing all user attributes.

optuna.study.create_study(storage=None, sampler=None, pruner=None, study_name=None, direction='minimize', load_if_exists=False)[source]¶

Create a new Study.

Parameters

storage –
Database URL. If this argument is set to None, in-memory storage is used, and the Study will not be persistent.

Note

When a database URL is passed, Optuna internally uses SQLAlchemy to handle the database. Please refer to SQLAlchemy’s document for further details. If you want to specify non-default options to SQLAlchemy Engine, you can instantiate RDBStorage with your desired options and pass it to the storage argument instead of a URL.
sampler – A sampler object that implements background algorithm for value suggestion. If None is specified, TPESampler is used as the default. See also samplers.
pruner – A pruner object that decides early stopping of unpromising trials. See also pruners.
study_name – Study’s name. If this argument is set to None, a unique name is generated automatically.
direction – Direction of optimization. Set minimize for minimization and maximize for maximization.
load_if_exists – Flag to control the behavior to handle a conflict of study names. In the case where a study named study_name already exists in the storage, a DuplicatedStudyError is raised if load_if_exists is set to False. Otherwise, the creation of the study is skipped, and the existing one is returned.

Returns

A Study object.

See also

The alias also exists as optuna.create_study().

optuna.study.load_study(study_name, storage, sampler=None, pruner=None)[source]¶

Load the existing Study that has the specified name.

Parameters

study_name – Study’s name. Each study has a unique name as an identifier.
storage – Database URL such as sqlite:///example.db. Please see also the documentation of create_study() for further details.
sampler – A sampler object that implements background algorithm for value suggestion. If None is specified, TPESampler is used as the default. See also samplers.
pruner – A pruner object that decides early stopping of unpromising trials. If None is specified, MedianPruner is used as the default. See also pruners.

See also

The alias also exists as optuna.load_study().

optuna.study.delete_study(study_name, storage)[source]¶

Delete a Study object.

Parameters

study_name – Study’s name.
storage – Database URL such as sqlite:///example.db. Please see also the documentation of create_study() for further details.

See also

The alias also exists as optuna.delete_study().

optuna.study.get_all_study_summaries(storage)[source]¶

Get all history of studies stored in a specified storage.

Parameters: storage – Database URL such as sqlite:///example.db. Please see also the documentation of create_study() for further details.
Returns: List of study history summarized as StudySummary objects.

See also

The alias also exists as optuna.get_all_study_summaries().

class optuna.study.StudyDirection[source]¶

Direction of a Study.

NOT_SET¶: Direction has not been set.

MINIMIZE¶: Study minimizes the objective function.

MAXIMIZE¶: Study maximizes the objective function.

class optuna.study.StudySummary(study_name: str, direction: optuna._study_direction.StudyDirection, best_trial: Optional[optuna.trial._frozen.FrozenTrial], user_attrs: Dict[str, Any], system_attrs: Dict[str, Any], n_trials: int, datetime_start: Optional[datetime.datetime], study_id: int)[source]¶

Basic attributes and aggregated results of a Study.

study_name¶: Name of the Study.

direction¶: StudyDirection of the Study.

best_trial¶: FrozenTrial with best objective value in the Study.

user_attrs¶: Dictionary that contains the attributes of the Study set with optuna.study.Study.set_user_attr().

system_attrs¶: Dictionary that contains the attributes of the Study internally set by Optuna.

n_trials¶: The number of trials ran in the Study.

datetime_start¶: Datetime where the Study started.

optuna.trial¶

class optuna.trial.Trial(study, trial_id)[source]¶

A trial is a process of evaluating an objective function.

This object is passed to an objective function and provides interfaces to get parameter suggestion, manage the trial’s state, and set/get user-defined attributes of the trial.

Note that the direct use of this constructor is not recommended. This object is seamlessly instantiated and passed to the objective function behind the optuna.study.Study.optimize() method; hence library users do not care about instantiation of this object.

Parameters

study – A Study object.
trial_id – A trial ID that is automatically generated.

property datetime_start¶

Return start datetime.

Returns: Datetime where the Trial started.

property distributions¶

Return distributions of parameters to be optimized.

Returns: A dictionary containing all distributions.

property number¶

Return trial’s number which is consecutive and unique in a study.

Returns: A trial number.

property params¶

Return parameters to be optimized.

Returns: A dictionary containing all parameters.

report(value, step)[source]¶

Report an objective function value for a given step.

The reported values are used by the pruners to determine whether this trial should be pruned.

See also

Please refer to BasePruner.

Note

The reported value is converted to float type by applying float() function internally. Thus, it accepts all float-like types (e.g., numpy.float32). If the conversion fails, a TypeError is raised.

Example

Report intermediate scores of SGDClassifier training.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import SGDClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    clf = SGDClassifier(random_state=0)
    for step in range(100):
        clf.partial_fit(X_train, y_train, np.unique(y))
        intermediate_value = clf.score(X_valid, y_valid)
        trial.report(intermediate_value, step=step)
        if trial.should_prune():
            raise optuna.TrialPruned()

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)

Parameters

value – A value returned from the objective function.
step – Step of the trial (e.g., Epoch of neural network training).

set_user_attr(key, value)[source]¶

Set user attributes to the trial.

The user attributes in the trial can be access via optuna.trial.Trial.user_attrs().

Example

Save fixed hyperparameters of neural network training.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y, random_state=0)

def objective(trial):
    trial.set_user_attr('BATCHSIZE', 128)
    momentum = trial.suggest_uniform('momentum', 0, 1.0)
    clf = MLPClassifier(hidden_layer_sizes=(100, 50),
                        batch_size=trial.user_attrs['BATCHSIZE'],
                        momentum=momentum, solver='sgd', random_state=0)
    clf.fit(X_train, y_train)

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)
assert 'BATCHSIZE' in study.best_trial.user_attrs.keys()
assert study.best_trial.user_attrs['BATCHSIZE'] == 128

Parameters

key – A key string of the attribute.
value – A value of the attribute. The value should be JSON serializable.

should_prune() → bool [source]¶

Suggest whether the trial should be pruned or not.

The suggestion is made by a pruning algorithm associated with the trial and is based on previously reported values. The algorithm can be specified when constructing a Study.

Note

If no values have been reported, the algorithm cannot make meaningful suggestions. Similarly, if this method is called multiple times with the exact same set of reported values, the suggestions will be the same.

See also

Please refer to the example code in optuna.trial.Trial.report().

Returns: A boolean value. If True, the trial should be pruned according to the configured pruning algorithm. Otherwise, the trial should continue.

suggest_categorical(name, choices)[source]¶

Suggest a value for the categorical parameter.

The value is sampled from choices.

Example

Suggest a kernel function of SVC.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    kernel = trial.suggest_categorical('kernel', ['linear', 'poly', 'rbf'])
    clf = SVC(kernel=kernel, gamma='scale', random_state=0)
    clf.fit(X_train, y_train)
    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)

Parameters

name – A parameter name.
choices – Parameter value candidates.

See also

CategoricalDistribution.

Returns: A suggested value.

suggest_discrete_uniform(name, low, high, q)[source]¶

Suggest a value for the discrete parameter.

The value is sampled from the range $[\mathsf{low}, \mathsf{high}]$, and the step of discretization is $q$. More specifically, this method returns one of the values in the sequence $\mathsf{low}, \mathsf{low} + q, \mathsf{low} + 2 q, \dots, \mathsf{low} + k q \le \mathsf{high}$, where $k$ denotes an integer. Note that $high$ may be changed due to round-off errors if $q$ is not an integer. Please check warning messages to find the changed values.

Example

Suggest a fraction of samples used for fitting the individual learners of GradientBoostingClassifier.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    subsample = trial.suggest_discrete_uniform('subsample', 0.1, 1.0, 0.1)
    clf = GradientBoostingClassifier(subsample=subsample, random_state=0)
    clf.fit(X_train, y_train)
    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)

Parameters

name – A parameter name.
low – Lower endpoint of the range of suggested values. low is included in the range.
high – Upper endpoint of the range of suggested values. high is included in the range.
q – A step of discretization.

Returns

A suggested float value.

suggest_float(name: str, low: float, high: float, *, step: Optional[float] = None, log: bool = False) → float [source]¶

Suggest a value for the floating point parameter.

Note that this is a wrapper method for suggest_uniform(), suggest_loguniform() and suggest_discrete_uniform().

New in version 1.3.0.

Example

Suggest a momentum, learning rate and scaling factor of learning rate for neural network training.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y, random_state=0)

def objective(trial):
    momentum = trial.suggest_float('momentum', 0.0, 1.0)
    learning_rate_init = trial.suggest_float('learning_rate_init',
                                             1e-5, 1e-3, log=True)
    power_t = trial.suggest_float('power_t', 0.2, 0.8, step=0.1)
    clf = MLPClassifier(hidden_layer_sizes=(100, 50), momentum=momentum,
                        learning_rate_init=learning_rate_init,
                        solver='sgd', random_state=0, power_t=power_t)
    clf.fit(X_train, y_train)

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)

Parameters

name – A parameter name.
low – Lower endpoint of the range of suggested values. low is included in the range.
high – Upper endpoint of the range of suggested values. high is excluded from the range.
step –
A step of discretization.

Note

The step and log arguments cannot be used at the same time. To set the step argument to a float number, set the log argument to False.
log –
A flag to sample the value from the log domain or not. If log is true, the value is sampled from the range in the log domain. Otherwise, the value is sampled from the range in the linear domain. See also suggest_uniform() and suggest_loguniform().

Note

The step and log arguments cannot be used at the same time. To set the log argument to True, set the step argument to None.

Raises

ValueError – If step is not None and log = True are specified.

Returns

A suggested float value.

suggest_int(name: str, low: int, high: int, step: int = 1, log: bool = False) → int [source]¶

Suggest a value for the integer parameter.

The value is sampled from the integers in $[\mathsf{low}, \mathsf{high}]$.

Example

Suggest the number of trees in RandomForestClassifier.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    n_estimators = trial.suggest_int('n_estimators', 50, 400)
    clf = RandomForestClassifier(n_estimators=n_estimators, random_state=0)
    clf.fit(X_train, y_train)
    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)

Parameters

name – A parameter name.
low – Lower endpoint of the range of suggested values. low is included in the range.
high – Upper endpoint of the range of suggested values. high is included in the range.
step –
A step of discretization.

Note

Note that $\mathsf{high}$ is modified if the range is not divisible by $\mathsf{step}$. Please check the warning messages to find the changed values.

Note

The method returns one of the values in the sequence $\mathsf{low}, \mathsf{low} + \mathsf{step}, \mathsf{low} + 2 * \mathsf{step}, \dots, \mathsf{low} + k * \mathsf{step} \le \mathsf{high}$, where $k$ denotes an integer.

Note

The step != 1 and log arguments cannot be used at the same time. To set the step argument $\mathsf{step} \ge 2$, set the log argument to False.
log –
A flag to sample the value from the log domain or not.

Note

If log is true, at first, the range of suggested values is divided into grid points of width 1. The range of suggested values is then converted to a log domain, from which a value is sampled. The uniformly sampled value is re-converted to the original domain and rounded to the nearest grid point that we just split, and the suggested value is determined. For example, if low = 2 and high = 8, then the range of suggested values is [2, 3, 4, 5, 6, 7, 8] and lower values tend to be more sampled than higher values.

Note

The step != 1 and log arguments cannot be used at the same time. To set the log argument to True, set the step argument to 1.

Raises

ValueError – If step != 1 and log = True are specified.

suggest_loguniform(name, low, high)[source]¶

Suggest a value for the continuous parameter.

The value is sampled from the range $[\mathsf{low}, \mathsf{high})$ in the log domain. When $\mathsf{low} = \mathsf{high}$, the value of $\mathsf{low}$ will be returned.

Example

Suggest penalty parameter C of SVC.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    c = trial.suggest_loguniform('c', 1e-5, 1e2)
    clf = SVC(C=c, gamma='scale', random_state=0)
    clf.fit(X_train, y_train)
    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)

Parameters

name – A parameter name.
low – Lower endpoint of the range of suggested values. low is included in the range.
high – Upper endpoint of the range of suggested values. high is excluded from the range.

Returns

A suggested float value.

suggest_uniform(name, low, high)[source]¶

Suggest a value for the continuous parameter.

The value is sampled from the range $[\mathsf{low}, \mathsf{high})$ in the linear domain. When $\mathsf{low} = \mathsf{high}$, the value of $\mathsf{low}$ will be returned.

Example

Suggest a momentum for neural network training.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

import optuna

X, y = load_iris(return_X_y=True)
X_train, X_valid, y_train, y_valid = train_test_split(X, y)

def objective(trial):
    momentum = trial.suggest_uniform('momentum', 0.0, 1.0)
    clf = MLPClassifier(hidden_layer_sizes=(100, 50), momentum=momentum,
                        solver='sgd', random_state=0)
    clf.fit(X_train, y_train)

    return clf.score(X_valid, y_valid)

study = optuna.create_study(direction='maximize')
study.optimize(objective, n_trials=3)

Parameters

name – A parameter name.
low – Lower endpoint of the range of suggested values. low is included in the range.
high – Upper endpoint of the range of suggested values. high is excluded from the range.

Returns

A suggested float value.

property user_attrs¶

Return user attributes.

Returns: A dictionary containing all user attributes.

class optuna.trial.FixedTrial(params, number=0)[source]¶

A trial class which suggests a fixed value for each parameter.

This object has the same methods as Trial, and it suggests pre-defined parameter values. The parameter values can be determined at the construction of the FixedTrial object. In contrast to Trial, FixedTrial does not depend on Study, and it is useful for deploying optimization results.

Example

Evaluate an objective function with parameter values given by a user.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -100, 100)
    y = trial.suggest_categorical('y', [-1, 0, 1])
    return x ** 2 + y

assert objective(optuna.trial.FixedTrial({'x': 1, 'y': 0})) == 1

Note

Please refer to Trial for details of methods and properties.

Parameters

params – A dictionary containing all parameters.
number – A trial number. Defaults to 0.

class optuna.trial.FrozenTrial(number, state, value, datetime_start, datetime_complete, params, distributions, user_attrs, system_attrs, intermediate_values, trial_id)[source]¶

Status and results of a Trial.

number¶: Unique and consecutive number of Trial for each Study. Note that this field uses zero-based numbering.

state¶: TrialState of the Trial.

value¶: Objective value of the Trial.

datetime_start¶: Datetime where the Trial started.

datetime_complete¶: Datetime where the Trial finished.

params¶: Dictionary that contains suggested parameters.

user_attrs¶: Dictionary that contains the attributes of the Trial set with optuna.trial.Trial.set_user_attr().

intermediate_values¶: Intermediate objective values set with optuna.trial.Trial.report().

property distributions¶: Dictionary that contains the distributions of params.

property duration¶

Return the elapsed time taken to complete the trial.

Returns: The duration.

class optuna.trial.TrialState[source]¶

State of a Trial.

RUNNING¶: The Trial is running.

COMPLETE¶: The Trial has been finished without any error.

PRUNED¶: The Trial has been pruned with TrialPruned.

FAIL¶: The Trial has failed due to an uncaught error.

optuna.visualization¶

Note

visualization module uses plotly to create figures, but JupyterLab cannot render them by default. Please follow this installation guide to show figures in JupyterLab.

optuna.visualization.plot_contour(study: optuna.study.Study, params: Optional[List[str]] = None) → go.Figure[source]¶

Plot the parameter relationship as contour plot in a study.

Note that, If a parameter contains missing values, a trial with missing values is not plotted.

Example

The following code snippet shows how to plot the parameter relationship as contour plot.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -100, 100)
    y = trial.suggest_categorical('y', [-1, 0, 1])
    return x ** 2 + y

study = optuna.create_study()
study.optimize(objective, n_trials=10)

optuna.visualization.plot_contour(study, params=['x', 'y'])

Parameters

study – A Study object whose trials are plotted for their objective values.
params – Parameter list to visualize. The default is all parameters.

Returns

A plotly.graph_objs.Figure object.

optuna.visualization.plot_intermediate_values(study: optuna.study.Study) → go.Figure[source]¶

Plot intermediate values of all trials in a study.

Example

The following code snippet shows how to plot intermediate values.

import optuna

def f(x):
    return (x - 2) ** 2

def df(x):
    return 2 * x - 4

def objective(trial):
    lr = trial.suggest_loguniform("lr", 1e-5, 1e-1)

    x = 3
    for step in range(128):
        y = f(x)

        trial.report(y, step=step)
        if trial.should_prune():
            raise optuna.TrialPruned()

        gy = df(x)
        x -= gy * lr

    return y

study = optuna.create_study()
study.optimize(objective, n_trials=16)

optuna.visualization.plot_intermediate_values(study)

Parameters: study – A Study object whose trials are plotted for their intermediate values.
Returns: A plotly.graph_objs.Figure object.

optuna.visualization.plot_optimization_history(study: optuna.study.Study) → go.Figure[source]¶

Plot optimization history of all trials in a study.

Example

The following code snippet shows how to plot optimization history.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -100, 100)
    y = trial.suggest_categorical('y', [-1, 0, 1])
    return x ** 2 + y

study = optuna.create_study()
study.optimize(objective, n_trials=10)

optuna.visualization.plot_optimization_history(study)

Parameters: study – A Study object whose trials are plotted for their objective values.
Returns: A plotly.graph_objs.Figure object.

optuna.visualization.plot_parallel_coordinate(study: optuna.study.Study, params: Optional[List[str]] = None) → go.Figure[source]¶

Plot the high-dimentional parameter relationships in a study.

Note that, If a parameter contains missing values, a trial with missing values is not plotted.

Example

The following code snippet shows how to plot the high-dimentional parameter relationships.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -100, 100)
    y = trial.suggest_categorical('y', [-1, 0, 1])
    return x ** 2 + y

study = optuna.create_study()
study.optimize(objective, n_trials=10)

optuna.visualization.plot_parallel_coordinate(study, params=['x', 'y'])

Parameters

study – A Study object whose trials are plotted for their objective values.
params – Parameter list to visualize. The default is all parameters.

Returns

A plotly.graph_objs.Figure object.

optuna.visualization.plot_param_importances(study: optuna.study.Study, evaluator: optuna.importance._base.BaseImportanceEvaluator = None, params: Optional[List[str]] = None) → go.Figure[source]¶

Plot hyperparameter importances.

Example

The following code snippet shows how to plot hyperparameter importances.

import optuna

def objective(trial):
    x = trial.suggest_int("x", 0, 2)
    y = trial.suggest_float("y", -1.0, 1.0)
    z = trial.suggest_float("z", 0.0, 1.5)
    return x ** 2 + y ** 3 - z ** 4

study = optuna.create_study(sampler=optuna.samplers.RandomSampler())
study.optimize(objective, n_trials=100)

optuna.visualization.plot_param_importances(study)

See also

This function visualizes the results of optuna.importance.get_param_importances().

Parameters

study – An optimized study.
evaluator – An importance evaluator object that specifies which algorithm to base the importance assessment on. Defaults to MeanDecreaseImpurityImportanceEvaluator.
params – A list of names of parameters to assess. If None, all parameters that are present in all of the completed trials are assessed.

Returns

A plotly.graph_objs.Figure object.

Note

Added in v1.5.0 as an experimental feature. The interface may change in newer versions without prior notice. See https://github.com/optuna/optuna/releases/tag/v1.5.0.

optuna.visualization.plot_slice(study: optuna.study.Study, params: Optional[List[str]] = None) → go.Figure[source]¶

Plot the parameter relationship as slice plot in a study.

Note that, If a parameter contains missing values, a trial with missing values is not plotted.

Example

The following code snippet shows how to plot the parameter relationship as slice plot.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -100, 100)
    y = trial.suggest_categorical('y', [-1, 0, 1])
    return x ** 2 + y

study = optuna.create_study()
study.optimize(objective, n_trials=10)

optuna.visualization.plot_slice(study, params=['x', 'y'])

Parameters

study – A Study object whose trials are plotted for their objective values.
params – Parameter list to visualize. The default is all parameters.

Returns

A plotly.graph_objs.Figure object.

optuna.visualization.is_available() → bool [source]¶

Returns whether visualization is available or not.

Note

visualization module depends on plotly version 4.0.0 or higher. If a supported version of plotly isn’t installed in your environment, this function will return False. In such case, please execute $ pip install -U plotly>=4.0.0 to install plotly.

Returns: True if visualization is available, False otherwise.

FAQ¶

Can I use Optuna with X? (where X is your favorite ML library)
How to define objective functions that have own arguments?
Can I use Optuna without remote RDB servers?
How can I save and resume studies?
How to suppress log messages of Optuna?
How to save machine learning models trained in objective functions?
How can I obtain reproducible optimization results?
How are exceptions from trials handled?
How are NaNs returned by trials handled?
What happens when I dynamically alter a search space?
How can I use two GPUs for evaluating two trials simultaneously?
How can I test my objective functions?
How do I avoid running out of memory (OOM) when optimizing studies?

Can I use Optuna with X? (where X is your favorite ML library)¶

Optuna is compatible with most ML libraries, and it’s easy to use Optuna with those. Please refer to examples.

How to define objective functions that have own arguments?¶

There are two ways to realize it.

First, callable classes can be used for that purpose as follows:

import optuna

class Objective(object):
    def __init__(self, min_x, max_x):
        # Hold this implementation specific arguments as the fields of the class.
        self.min_x = min_x
        self.max_x = max_x

    def __call__(self, trial):
        # Calculate an objective value by using the extra arguments.
        x = trial.suggest_uniform('x', self.min_x, self.max_x)
        return (x - 2) ** 2

# Execute an optimization by using an `Objective` instance.
study = optuna.create_study()
study.optimize(Objective(-100, 100), n_trials=100)

Second, you can use lambda or functools.partial for creating functions (closures) that hold extra arguments. Below is an example that uses lambda:

import optuna

# Objective function that takes three arguments.
def objective(trial, min_x, max_x):
    x = trial.suggest_uniform('x', min_x, max_x)
    return (x - 2) ** 2

# Extra arguments.
min_x = -100
max_x = 100

# Execute an optimization by using the above objective function wrapped by `lambda`.
study = optuna.create_study()
study.optimize(lambda trial: objective(trial, min_x, max_x), n_trials=100)

Please also refer to sklearn_addtitional_args.py example.

Can I use Optuna without remote RDB servers?¶

Yes, it’s possible.

In the simplest form, Optuna works with in-memory storage:

study = optuna.create_study()
study.optimize(objective)

If you want to save and resume studies, it’s handy to use SQLite as the local storage:

study = optuna.create_study(study_name='foo_study', storage='sqlite:///example.db')
study.optimize(objective)  # The state of `study` will be persisted to the local SQLite file.

Please see Saving/Resuming Study with RDB Backend for more details.

How can I save and resume studies?¶

There are two ways of persisting studies, which depends if you are using in-memory storage (default) or remote databases (RDB). In-memory studies can be saved and loaded like usual Python objects using pickle or joblib. For example, using joblib:

study = optuna.create_study()
joblib.dump(study, 'study.pkl')

And to resume the study:

study = joblib.load('study.pkl')
print('Best trial until now:')
print(' Value: ', study.best_trial.value)
print(' Params: ')
for key, value in study.best_trial.params.items():
    print(f'    {key}: {value}')

If you are using RDBs, see Saving/Resuming Study with RDB Backend for more details.

How to suppress log messages of Optuna?¶

By default, Optuna shows log messages at the optuna.logging.INFO level. You can change logging levels by using optuna.logging.set_verbosity().

For instance, you can stop showing each trial result as follows:

optuna.logging.set_verbosity(optuna.logging.WARNING)

study = optuna.create_study()
study.optimize(objective)
# Logs like '[I 2018-12-05 11:41:42,324] Finished a trial resulted in value:...' are disabled.

Please refer to optuna.logging for further details.

How to save machine learning models trained in objective functions?¶

Optuna saves hyperparameter values with its corresponding objective value to storage, but it discards intermediate objects such as machine learning models and neural network weights. To save models or weights, please use features of the machine learning library you used.

We recommend saving optuna.trial.Trial.number with a model in order to identify its corresponding trial. For example, you can save SVM models trained in the objective function as follows:

def objective(trial):
    svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
    clf = sklearn.svm.SVC(C=svc_c)
    clf.fit(X_train, y_train)

    # Save a trained model to a file.
    with open('{}.pickle'.format(trial.number), 'wb') as fout:
        pickle.dump(clf, fout)
    return 1.0 - accuracy_score(y_valid, clf.predict(X_valid))


study = optuna.create_study()
study.optimize(objective, n_trials=100)

# Load the best model.
with open('{}.pickle'.format(study.best_trial.number), 'rb') as fin:
    best_clf = pickle.load(fin)
print(accuracy_score(y_valid, best_clf.predict(X_valid)))

How can I obtain reproducible optimization results?¶

To make the parameters suggested by Optuna reproducible, you can specify a fixed random seed via seed argument of RandomSampler or TPESampler as follows:

sampler = TPESampler(seed=10)  # Make the sampler behave in a deterministic way.
study = optuna.create_study(sampler=sampler)
study.optimize(objective)

However, there are two caveats.

First, when optimizing a study in distributed or parallel mode, there is inherent non-determinism. Thus it is very difficult to reproduce the same results in such condition. We recommend executing optimization of a study sequentially if you would like to reproduce the result.

Second, if your objective function behaves in a non-deterministic way (i.e., it does not return the same value even if the same parameters were suggested), you cannot reproduce an optimization. To deal with this problem, please set an option (e.g., random seed) to make the behavior deterministic if your optimization target (e.g., an ML library) provides it.

How are exceptions from trials handled?¶

Trials that raise exceptions without catching them will be treated as failures, i.e. with the FAIL status.

By default, all exceptions except TrialPruned raised in objective functions are propagated to the caller of optimize(). In other words, studies are aborted when such exceptions are raised. It might be desirable to continue a study with the remaining trials. To do so, you can specify in optimize() which exception types to catch using the catch argument. Exceptions of these types are caught inside the study and will not propagate further.

You can find the failed trials in log messages.

[W 2018-12-07 16:38:36,889] Setting status of trial#0 as TrialState.FAIL because of \
the following error: ValueError('A sample error in objective.')

You can also find the failed trials by checking the trial states as follows:

study.trials_dataframe()

number	state	value	…	params	system_attrs
0	TrialState.FAIL		…	0	Setting status of trial#0 as TrialState.FAIL because of the following error: ValueError(‘A test error in objective.’)
1	TrialState.COMPLETE	1269	…	1

See also

The catch argument in optimize().

How are NaNs returned by trials handled?¶

Trials that return NaN (float('nan')) are treated as failures, but they will not abort studies.

Trials which return NaN are shown as follows:

[W 2018-12-07 16:41:59,000] Setting status of trial#2 as TrialState.FAIL because the \
objective function returned nan.

What happens when I dynamically alter a search space?¶

Since parameters search spaces are specified in each call to the suggestion API, e.g. suggest_uniform() and suggest_int(), it is possible to in a single study alter the range by sampling parameters from different search spaces in different trials. The behavior when altered is defined by each sampler individually.

Note

Discussion about the TPE sampler. https://github.com/optuna/optuna/issues/822

How can I use two GPUs for evaluating two trials simultaneously?¶

If your optimization target supports GPU (CUDA) acceleration and you want to specify which GPU is used, the easiest way is to set CUDA_VISIBLE_DEVICES environment variable:

# On a terminal.
#
# Specify to use the first GPU, and run an optimization.
$ export CUDA_VISIBLE_DEVICES=0
$ optuna study optimize foo.py objective --study-name foo --storage sqlite:///example.db

# On another terminal.
#
# Specify to use the second GPU, and run another optimization.
$ export CUDA_VISIBLE_DEVICES=1
$ optuna study optimize bar.py objective --study-name bar --storage sqlite:///example.db

Please refer to CUDA C Programming Guide for further details.

How can I test my objective functions?¶

When you test objective functions, you may prefer fixed parameter values to sampled ones. In that case, you can use FixedTrial, which suggests fixed parameter values based on a given dictionary of parameters. For instance, you can input arbitrary values of $x$ and $y$ to the objective function $x + y$ as follows:

def objective(trial):
    x = trial.suggest_uniform('x', -1.0, 1.0)
    y = trial.suggest_int('y', -5, 5)
    return x + y

objective(FixedTrial({'x': 1.0, 'y': -1}))  # 0.0
objective(FixedTrial({'x': -1.0, 'y': -4}))  # -5.0

Using FixedTrial, you can write unit tests as follows:

# A test function of pytest
def test_objective():
    assert 1.0 == objective(FixedTrial({'x': 1.0, 'y': 0}))
    assert -1.0 == objective(FixedTrial({'x': 0.0, 'y': -1}))
    assert 0.0 == objective(FixedTrial({'x': -1.0, 'y': 1}))

How do I avoid running out of memory (OOM) when optimizing studies?¶

If the memory footprint increases as you run more trials, try to periodically run the garbage collector. Specify gc_after_trial to True when calling optimize() or call gc.collect() inside a callback.

def objective(trial):
    x = trial.suggest_uniform('x', -1.0, 1.0)
    y = trial.suggest_int('y', -5, 5)
    return x + y

study = optuna.create_study()
study.optimize(objective, n_trials=10, gc_after_trial=True)

# `gc_after_trial=True` is more or less identical to the following.
study.optimize(objective, n_trials=10, callbacks=[lambda study, trial: gc.collect()])

There is a performance trade-off for running the garbage collector, which could be non-negligible depending on how fast your objective function otherwise is. Therefore, gc_after_trial is False by default. Note that the above examples are similar to running the garbage collector inside the objective function, except for the fact that gc.collect() is called even when errors, including TrialPruned are raised.

Note

ChainerMNStudy does currently not provide gc_after_trial nor callbacks for optimize(). When using this class, you will have to call the garbage collector inside the objective function.

Optuna: A hyperparameter optimization framework¶

Key Features¶

Basic Concepts¶

Communication¶

Contribution¶

License¶

Reference¶

Installation¶

Tutorial¶

First Optimization¶

Quadratic Function Example¶

Study Object¶

Advanced Configurations¶

Defining Parameter Spaces¶

Branches and Loops¶

Note on the Number of Parameters¶

Arguments for Study.optimize¶

Saving/Resuming Study with RDB Backend¶

New Study¶

Resume Study¶

Experimental History¶

Distributed Optimization¶

Command-Line Interface¶

User Attributes¶

Adding User Attributes to Studies¶

Adding User Attributes to Trials¶

Pruning Unpromising Trials¶

Activating Pruners¶

Integration Modules for Pruning¶

User-Defined Sampler¶

Overview of Sampler¶

An Example: Implementing SimulatedAnnealingSampler¶

API Reference¶

optuna¶

optuna.cli¶

create-study¶

dashboard¶

delete-study¶

storage upgrade¶

studies¶

study optimize¶

study set-user-attr¶

optuna.distributions¶

optuna.exceptions¶

optuna.importance¶

optuna.integration¶

optuna.logging¶

optuna.multi_objective¶

optuna.multi_objective.samplers¶

optuna.multi_objective.study¶

optuna.multi_objective.trial¶

optuna.multi_objective.visualization¶

optuna.pruners¶

optuna.samplers¶

optuna.storages¶

optuna.structs¶

optuna.study¶

optuna.trial¶

optuna.visualization¶

FAQ¶

Indices and tables¶