Optuna: A hyperparameter optimization framework¶

Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, define-by-run style user API. Thanks to our define-by-run API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters.

Key Features¶

Optuna has modern functionalities as follows:

Basic Concepts¶

We use the terms study and trial as follows:

Study: optimization based on an objective function
Trial: a single execution of the objective function

Please refer to sample code below. The goal of a study is to find out the optimal set of hyperparameter values (e.g., classifier and svm_c) through multiple trials (e.g., n_trials=100). Optuna is a framework designed for the automation and the acceleration of the optimization studies.

import ...

# Define an objective function to be minimized.
def objective(trial):

    # Invoke suggest methods of a Trial object to generate hyperparameters.
    regressor_name = trial.suggest_categorical('classifier', ['SVR', 'RandomForest'])
    if regressor_name == 'SVR':
        svr_c = trial.suggest_loguniform('svr_c', 1e-10, 1e10)
        regressor_obj = sklearn.svm.SVR(C=svr_c)
    else:
        rf_max_depth = trial.suggest_int('rf_max_depth', 2, 32)
        regressor_obj = sklearn.ensemble.RandomForestRegressor(max_depth=rf_max_depth)

    X, y = sklearn.datasets.load_boston(return_X_y=True)
    X_train, X_val, y_train, y_val = sklearn.model_selection.train_test_split(X, y, random_state=0)

    regressor_obj.fit(X_train, y_train)
    y_pred = regressor_obj.predict(X_val)

    error = sklearn.metrics.mean_squared_error(y_val, y_pred)

    return error  # An objective value linked with the Trial object.

study = optuna.create_study()  # Create a new study.
study.optimize(objective, n_trials=100)  # Invoke optimization of the objective function.

Communication¶

GitHub Issues for bug reports, feature requests and questions.
Gitter for interactive chat with developers.
Stack Overflow for questions.

Contribution¶

Any contributions to Optuna are welcome! When you send a pull request, please follow the contribution guide.

License¶

MIT License (see LICENSE).

Reference¶

Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In KDD (arXiv).

Installation¶

Optuna supports Python 3.5 or newer.

We recommend to install Optuna via pip:

$ pip install optuna

You can also install the development version of Optuna from master branch of Git repository:

$ pip install git+https://github.com/optuna/optuna.git

You can also install Optuna via conda:

$ conda install -c conda-forge optuna

Tutorial¶

First Optimization¶

Quadratic Function Example¶

Usually, Optuna is used to optimize hyper-parameters, but as an example, let us directly optimize a quadratic function in an IPython shell.

import optuna

The objective function is what will be optimized.

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

This function returns the value of \((x - 2)^2\). Our goal is to find the value of x that minimizes the output of the objective function. This is the “optimization.” During the optimization, Optuna repeatedly calls and evaluates the objective function with different values of x.

A Trial object corresponds to a single execution of the objective function and is internally instantiated upon each invocation of the function.

The suggest APIs (for example, suggest_uniform()) are called inside the objective function to obtain parameters for a trial. suggest_uniform() selects parameters uniformly within the range provided. In our example, from -10 to 10.

To start the optimization, we create a study object and pass the objective function to method optimize() as follows.

study = optuna.create_study()
study.optimize(objective, n_trials=100)

Out:

[I 2020-04-08 10:42:09,028] Trial 0 finished with value: 25.77382032395108 with parameters: {'x': 7.076792326257898}. Best is trial 0 with value: 25.77382032395108.
[I 2020-04-08 10:42:09,064] Trial 1 finished with value: 1.5189812248635903 with parameters: {'x': 0.7675304365366298}. Best is trial 1 with value: 1.5189812248635903.
[I 2020-04-08 10:42:09,106] Trial 2 finished with value: 34.4074691838153 with parameters: {'x': -3.865788027521562}. Best is trial 1 with value: 1.5189812248635903.
[I 2020-04-08 10:42:09,145] Trial 3 finished with value: 3.3601305753722657 with parameters: {'x': 3.8330658949891205}. Best is trial 1 with value: 1.5189812248635903.
[I 2020-04-08 10:42:09,185] Trial 4 finished with value: 61.16797535698886 with parameters: {'x': -5.820995803412048}. Best is trial 1 with value: 1.5189812248635903.
[I 2020-04-08 10:42:09,228] Trial 5 finished with value: 90.08665552769618 with parameters: {'x': -7.491399028999686}. Best is trial 1 with value: 1.5189812248635903.
[I 2020-04-08 10:42:09,274] Trial 6 finished with value: 25.254236332163032 with parameters: {'x': 7.025359323686519}. Best is trial 1 with value: 1.5189812248635903.
...
[I 2020-04-08 10:42:14,237] Trial 99 finished with value: 0.5227007740782738 with parameters: {'x': 2.7229804797352926}. Best is trial 67 with value: 2.916284393762304e-06.

You can get the best parameter as follows.

study.best_params

Out:

{'x': 2.001707713205946}

We can see that Optuna found the best x value 2.001707713205946, which is close to the optimal value of 2.

Note

When used to search for hyper-parameters in machine learning, usually the objective function would return the loss or accuracy of the model.

Study Object¶

Let us clarify the terminology in Optuna as follows:

Trial: A single call of the objective function
Study: An optimization session, which is a set of trials
Parameter: A variable whose value is to be optimized, such as x in the above example

In Optuna, we use the study object to manage optimization. Method create_study() returns a study object. A study object has useful properties for analyzing the optimization outcome.

To get the best parameter:

study.best_params

Out:

{'x': 2.001707713205946}

To get the best value:

study.best_value

Out:

2.916284393762304e-06

To get the best trial:

study.best_trial

Out:

FrozenTrial(number=67, value=2.916284393762304e-06, datetime_start=datetime.datetime(2020, 4, 8, 10, 42, 12, 595884), datetime_complete=datetime.datetime(2020, 4, 8, 10, 42, 12, 639969), params={'x': 2.001707713205946}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=67, state=TrialState.COMPLETE)

To get all trials:

study.trials

Out:

[FrozenTrial(number=0, value=25.77382032395108, datetime_start=datetime.datetime(2020, 4, 8, 10, 42, 8, 987277), datetime_complete=datetime.datetime(2020, 4, 8, 10, 42, 9, 27959), params={'x': 7.076792326257898}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=0, state=TrialState.COMPLETE),
 ...
 user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=99, state=TrialState.COMPLETE)]

To get the number of trials:

len(study.trials)

Out:

By executing optimize() again, we can continue the optimization.

study.optimize(objective, n_trials=100)

To get the updated number of trials:

len(study.trials)

Out:

Advanced Configurations¶

Defining Parameter Spaces¶

Optuna supports five kinds of parameters.

def objective(trial):
    # Categorical parameter
    optimizer = trial.suggest_categorical('optimizer', ['MomentumSGD', 'Adam'])

    # Int parameter
    num_layers = trial.suggest_int('num_layers', 1, 3)

    # Uniform parameter
    dropout_rate = trial.suggest_uniform('dropout_rate', 0.0, 1.0)

    # Loguniform parameter
    learning_rate = trial.suggest_loguniform('learning_rate', 1e-5, 1e-2)

    # Discrete-uniform parameter
    drop_path_rate = trial.suggest_discrete_uniform('drop_path_rate', 0.0, 1.0, 0.1)

    ...

Branches and Loops¶

You can use branches or loops depending on the parameter values.

def objective(trial):
    classifier_name = trial.suggest_categorical('classifier', ['SVC', 'RandomForest'])
    if classifier_name == 'SVC':
        svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
        classifier_obj = sklearn.svm.SVC(C=svc_c)
    else:
        rf_max_depth = int(trial.suggest_loguniform('rf_max_depth', 2, 32))
        classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth)

    ...

def create_model(trial):
    n_layers = trial.suggest_int('n_layers', 1, 3)

    layers = []
    for i in range(n_layers):
        n_units = int(trial.suggest_loguniform('n_units_l{}'.format(i), 4, 128))
        layers.append(L.Linear(None, n_units))
        layers.append(F.relu)
    layers.append(L.Linear(None, 10))

    return chainer.Sequential(*layers)

Please also refer to examples.

Note on the Number of Parameters¶

The difficulty of optimization increases roughly exponentially with regard to the number of parameters. That is, the number of necessary trials increases exponentially when you increase the number of parameters, so it is recommended to not add unimportant parameters.

Arguments for Study.optimize¶

The method optimize() (and optuna study optimize CLI command as well) has several useful options such as timeout. For details, please refer to the API reference for optimize().

FYI: If you give neither n_trials nor timeout options, the optimization continues until it receives a termination signal such as Ctrl+C or SIGTERM. This is useful for use cases such as when it is hard to estimate the computational costs required to optimize your objective function.

Saving/Resuming Study with RDB Backend¶

An RDB backend enables persistent experiments (i.e., to save and resume a study) as well as access to history of studies. In addition, we can run multi-node optimization tasks with this feature, which is described in Distributed Optimization.

In this section, let’s try simple examples running on a local environment with SQLite DB.

Note

You can also utilize other RDB backends, e.g., PostgreSQL or MySQL, by setting the storage argument to the DB’s URL. Please refer to SQLAlchemy’s document for how to set up the URL.

New Study¶

We can create a persistent study by calling create_study() function as follows. An SQLite file example.db is automatically initialized with a new study record.

import optuna
study_name = 'example-study'  # Unique identifier of the study.
study = optuna.create_study(study_name=study_name, storage='sqlite:///example.db')

To run a study, call optimize() method passing an objective function.

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

study.optimize(objective, n_trials=3)

Resume Study¶

To resume a study, instantiate a Study object passing the study name example-study and the DB URL sqlite:///example.db.

study = optuna.create_study(study_name='example-study', storage='sqlite:///example.db', load_if_exists=True)
study.optimize(objective, n_trials=3)

Experimental History¶

We can access histories of studies and trials via the Study class. For example, we can get all trials of example-study as:

import optuna
study = optuna.create_study(study_name='example-study', storage='sqlite:///example.db', load_if_exists=True)
df = study.trials_dataframe(attrs=('number', 'value', 'params', 'state'))

The method trials_dataframe() returns a pandas dataframe like:

print(df)

Out:

   number       value  params_x     state
     0   25.301959 -3.030105  COMPLETE
     1    1.406223  0.814157  COMPLETE
     2   44.010366 -4.634031  COMPLETE
     3   55.872181  9.474770  COMPLETE
     4  113.039223 -8.631991  COMPLETE
     5   57.319570  9.570969  COMPLETE

A Study object also provides properties such as trials, best_value, best_params (see also First Optimization).

study.best_params  # Get best parameters for the objective function.
study.best_value  # Get best objective value.
study.best_trial  # Get best trial's information.
study.trials  # Get all trials' information.

Distributed Optimization¶

There is no complicated setup but just sharing the same study name among nodes/processes.

First, create a shared study using optuna create-study command (or using optuna.create_study() in a Python script).

$ optuna create-study --study-name "distributed-example" --storage "sqlite:///example.db"
[I 2018-10-31 18:21:57,885] A new study created with name: distributed-example

Then, write an optimization script. Let’s assume that foo.py contains the following code.

import optuna

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

if __name__ == '__main__':
    study = optuna.load_study(study_name='distributed-example', storage='sqlite:///example.db')
    study.optimize(objective, n_trials=100)

Finally, run the shared study from multiple processes. For example, run Process 1 in a terminal, and do Process 2 in another one. They get parameter suggestions based on shared trials’ history.

Process 1:

$ python foo.py
[I 2018-10-31 18:46:44,308] Finished a trial resulted in value: 1.1097007755908204. Current best value is 0.00020881104123229936 with parameters: {'x': 2.014450295541348}.
[I 2018-10-31 18:46:44,361] Finished a trial resulted in value: 0.5186699439824186. Current best value is 0.00020881104123229936 with parameters: {'x': 2.014450295541348}.
...

Process 2 (the same command as process 1):

$ python foo.py
[I 2018-10-31 18:47:02,912] Finished a trial resulted in value: 29.821448668796563. Current best value is 0.00020881104123229936 with parameters: {'x': 2.014450295541348}.
[I 2018-10-31 18:47:02,968] Finished a trial resulted in value: 0.7962498978463782. Current best value is 0.00020881104123229936 with parameters: {'x': 2.014450295541348}.
...

Note

We do not recommend SQLite for large scale distributed optimizations because it may cause serious performance issues. Please consider to use another database engine like PostgreSQL or MySQL.

Note

Please avoid putting the SQLite database on NFS when running distributed optimizations. See also: https://www.sqlite.org/faq.html#q5

Command-Line Interface¶

Command	Description
create-study	Create a new study.
delete-study	Delete a specified study.
dashboard	Launch web dashboard (beta).
storage upgrade	Upgrade the schema of a storage.
studies	Show a list of studies.
study optimize	Start optimization of a study.
study set-user-attr	Set a user attribute to a study.

Optuna provides command-line interface as shown in the above table.

Let us assume you are not in IPython shell and writing Python script files instead. It is totally fine to write scripts like the following:

import optuna


def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2


if __name__ == '__main__':
    study = optuna.create_study()
    study.optimize(objective, n_trials=100)
    print('Best value: {} (params: {})\n'.format(study.best_value, study.best_params))

However, we can reduce boilerplate codes by using our optuna command. Let us assume that foo.py contains only the following code.

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

Even so, we can invoke the optimization as follows. (Don’t care about --storage sqlite:///example.db for now, which is described in Saving/Resuming Study with RDB Backend.)

$ cat foo.py
def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    return (x - 2) ** 2

$ STUDY_NAME=`optuna create-study --storage sqlite:///example.db`
$ optuna study optimize foo.py objective --n-trials=100 --storage sqlite:///example.db --study-name $STUDY_NAME
[I 2018-05-09 10:40:25,196] Finished a trial resulted in value: 54.353767789264026. Current best value is 54.353767789264026 with parameters: {'x': -5.372500782588228}.
[I 2018-05-09 10:40:25,197] Finished a trial resulted in value: 15.784266965526376. Current best value is 15.784266965526376 with parameters: {'x': 5.972941852774387}.
...
[I 2018-05-09 10:40:26,204] Finished a trial resulted in value: 14.704254135013741. Current best value is 2.280758099793617e-06 with parameters: {'x': 1.9984897821018828}.

Please note that foo.py only contains the definition of the objective function. By giving the script file name and the method name of objective function to optuna study optimize command, we can invoke the optimization.

User Attributes¶

This feature is to annotate experiments with user-defined attributes.

Adding User Attributes to Studies¶

A Study object provides set_user_attr() method to register a pair of key and value as an user-defined attribute. A key is supposed to be a str, and a value be any object serializable with json.dumps.

import optuna
study = optuna.create_study(storage='sqlite:///example.db')
study.set_user_attr('contributors', ['Akiba', 'Sano'])
study.set_user_attr('dataset', 'MNIST')

We can access annotated attributes with user_attr property.

study.user_attrs  # {'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'}

StudySummary object, which can be retrieved by get_all_study_summaries(), also contains user-defined attributes.

study_summaries = optuna.get_all_study_summaries('sqlite:///example.db')
study_summaries[0].user_attrs  # {'contributors': ['Akiba', 'Sano'], 'dataset': 'MNIST'}

Adding User Attributes to Trials¶

As with Study, a Trial object provides set_user_attr() method. Attributes are set inside an objective function.

def objective(trial):
    iris = sklearn.datasets.load_iris()
    x, y = iris.data, iris.target

    svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
    clf = sklearn.svm.SVC(C=svc_c)
    accuracy = sklearn.model_selection.cross_val_score(clf, x, y).mean()

    trial.set_user_attr('accuracy', accuracy)

    return 1.0 - accuracy  # return error for minimization

We can access annotated attributes as:

study.trials[0].user_attrs  # {'accuracy': 0.83}

Note that, in this example, the attribute is not annotated to a Study but a single Trial.

Pruning Unpromising Trials¶

This feature automatically stops unpromising trials at the early stages of the training (a.k.a., automated early-stopping). Optuna provides interfaces to concisely implement the pruning mechanism in iterative training algorithms.

Activating Pruners¶

To turn on the pruning feature, you need to call report() and should_prune() after each step of the iterative training. report() periodically monitors the intermediate objective values. should_prune() decides termination of the trial that does not meet a predefined condition.

"""filename: prune.py"""

import sklearn.datasets
import sklearn.linear_model
import sklearn.model_selection

import optuna

def objective(trial):
    iris = sklearn.datasets.load_iris()
    classes = list(set(iris.target))
    train_x, valid_x, train_y, valid_y = \
        sklearn.model_selection.train_test_split(iris.data, iris.target, test_size=0.25, random_state=0)

    alpha = trial.suggest_loguniform('alpha', 1e-5, 1e-1)
    clf = sklearn.linear_model.SGDClassifier(alpha=alpha)

    for step in range(100):
        clf.partial_fit(train_x, train_y, classes=classes)

        # Report intermediate objective value.
        intermediate_value = 1.0 - clf.score(valid_x, valid_y)
        trial.report(intermediate_value, step)

        # Handle pruning based on the intermediate value.
        if trial.should_prune():
            raise optuna.TrialPruned()

    return 1.0 - clf.score(valid_x, valid_y)

# Set up the median stopping rule as the pruning condition.
study = optuna.create_study(pruner=optuna.pruners.MedianPruner())
study.optimize(objective, n_trials=20)

Executing the script above:

$ python prune.py
[I 2020-06-12 16:54:23,876] Trial 0 finished with value: 0.3157894736842105 and parameters: {'alpha': 0.00181467547181131}. Best is trial 0 with value: 0.3157894736842105.
[I 2020-06-12 16:54:23,981] Trial 1 finished with value: 0.07894736842105265 and parameters: {'alpha': 0.015378744419287613}. Best is trial 1 with value: 0.07894736842105265.
[I 2020-06-12 16:54:24,083] Trial 2 finished with value: 0.21052631578947367 and parameters: {'alpha': 0.04089428832878595}. Best is trial 1 with value: 0.07894736842105265.
[I 2020-06-12 16:54:24,185] Trial 3 finished with value: 0.052631578947368474 and parameters: {'alpha': 0.004018735937374473}. Best is trial 3 with value: 0.052631578947368474.
[I 2020-06-12 16:54:24,303] Trial 4 finished with value: 0.07894736842105265 and parameters: {'alpha': 2.805688697062864e-05}. Best is trial 3 with value: 0.052631578947368474.
[I 2020-06-12 16:54:24,315] Trial 5 pruned.
[I 2020-06-12 16:54:24,355] Trial 6 pruned.
[I 2020-06-12 16:54:24,511] Trial 7 finished with value: 0.052631578947368474 and parameters: {'alpha': 2.243775785299103e-05}. Best is trial 3 with value: 0.052631578947368474.
[I 2020-06-12 16:54:24,625] Trial 8 finished with value: 0.1842105263157895 and parameters: {'alpha': 0.007021209286214553}. Best is trial 3 with value: 0.052631578947368474.
[I 2020-06-12 16:54:24,629] Trial 9 pruned.
...

Trial 5 pruned., etc. in the log messages means several trials were stopped before they finished all of the iterations.

Integration Modules for Pruning¶

To implement pruning mechanism in much simpler forms, Optuna provides integration modules for the following libraries.

XGBoost: optuna.integration.XGBoostPruningCallback
LightGBM: optuna.integration.LightGBMPruningCallback
Chainer: optuna.integration.ChainerPruningExtension
Keras: optuna.integration.KerasPruningCallback
TensorFlow optuna.integration.TensorFlowPruningHook
tf.keras optuna.integration.TFKerasPruningCallback
MXNet optuna.integration.MXNetPruningCallback
PyTorch Ignite optuna.integration.PyTorchIgnitePruningHandler
PyTorch Lightning optuna.integration.PyTorchLightningPruningCallback
FastAI optuna.integration.FastAIPruningCallback

For example, XGBoostPruningCallback introduces pruning without directly changing the logic of training iteration. (See also example for the entire script.)

pruning_callback = optuna.integration.XGBoostPruningCallback(trial, 'validation-error')
bst = xgb.train(param, dtrain, evals=[(dvalid, 'validation')], callbacks=[pruning_callback])

User-Defined Sampler¶

Thanks to user-defined samplers, you can:

experiment your own sampling algorithms,
implement task-specific algorithms to refine the optimization performance, or
wrap other optimization libraries to integrate them into Optuna pipelines (e.g., SkoptSampler).

This section describes the internal behavior of sampler classes and shows an example of implementing a user-defined sampler.

Overview of Sampler¶

A sampler has the responsibility to determine the parameter values to be evaluated in a trial. When a suggest API (e.g., suggest_uniform()) is called inside an objective function, the corresponding distribution object (e.g., UniformDistribution) is created internally. A sampler samples a parameter value from the distribution. The sampled value is returned to the caller of the suggest API and evaluated in the objective function.

To create a new sampler, you need to define a class that inherits BaseSampler. The base class has three abstract methods; infer_relative_search_space(), sample_relative(), and sample_independent().

As the method names imply, Optuna supports two types of sampling: one is relative sampling that can consider the correlation of the parameters in a trial, and the other is independent sampling that samples each parameter independently.

At the beginning of a trial, infer_relative_search_space() is called to provide the relative search space for the trial. Then, sample_relative() is invoked to sample relative parameters from the search space. During the execution of the objective function, sample_independent() is used to sample parameters that don’t belong to the relative search space.

Note

Please refer to the document of BaseSampler for further details.

An Example: Implementing SimulatedAnnealingSampler¶

For example, the following code defines a sampler based on Simulated Annealing (SA):

import numpy as np
import optuna


class SimulatedAnnealingSampler(optuna.samplers.BaseSampler):
    def __init__(self, temperature=100):
        self._rng = np.random.RandomState()
        self._temperature = temperature  # Current temperature.
        self._current_trial = None  # Current state.

    def sample_relative(self, study, trial, search_space):
        if search_space == {}:
            return {}

        #
        # An implementation of SA algorithm.
        #

        # Calculate transition probability.
        prev_trial = study.trials[-2]
        if self._current_trial is None or prev_trial.value <= self._current_trial.value:
            probability = 1.0
        else:
            probability = np.exp((self._current_trial.value - prev_trial.value) / self._temperature)
        self._temperature *= 0.9  # Decrease temperature.

        # Transit the current state if the previous result is accepted.
        if self._rng.uniform(0, 1) < probability:
            self._current_trial = prev_trial

        # Sample parameters from the neighborhood of the current point.
        #
        # The sampled parameters will be used during the next execution of
        # the objective function passed to the study.
        params = {}
        for param_name, param_distribution in search_space.items():
            if not isinstance(param_distribution, optuna.distributions.UniformDistribution):
                raise NotImplementedError('Only suggest_uniform() is supported')

            current_value = self._current_trial.params[param_name]
            width = (param_distribution.high - param_distribution.low) * 0.1
            neighbor_low = max(current_value - width, param_distribution.low)
            neighbor_high = min(current_value + width, param_distribution.high)
            params[param_name] = self._rng.uniform(neighbor_low, neighbor_high)

        return params

    #
    # The rest is boilerplate code and unrelated to SA algorithm.
    #
    def infer_relative_search_space(self, study, trial):
        return optuna.samplers.intersection_search_space(study)

    def sample_independent(self, study, trial, param_name, param_distribution):
        independent_sampler = optuna.samplers.RandomSampler()
        return independent_sampler.sample_independent(study, trial, param_name, param_distribution)

Note

In favor of code simplicity, the above implementation doesn’t support some features (e.g., maximization). If you’re interested in how to support those features, please see examples/samplers/simulated_annealing.py.

You can use SimulatedAnnealingSampler in the same way as built-in samplers as follows:

def objective(trial):
    x = trial.suggest_uniform('x', -10, 10)
    y = trial.suggest_uniform('y', -5, 5)
    return x**2 + y

sampler = SimulatedAnnealingSampler()
study = optuna.create_study(sampler=sampler)
study.optimize(objective, n_trials=100)

In this optimization, the values of x and y parameters are sampled by using SimulatedAnnealingSampler.sample_relative method.

Note

Strictly speaking, in the first trial, SimulatedAnnealingSampler.sample_independent method is used to sample parameter values. Because intersection_search_space() used in SimulatedAnnealingSampler.infer_relative_search_space cannot infer the search space if there are no complete trials.

API Reference¶

optuna¶

`optuna.create_study`	Create a new `Study`.
`optuna.load_study`	Load the existing `Study` that has the specified name.
`optuna.delete_study`	Delete a `Study` object.
`optuna.get_all_study_summaries`	Get all history of studies stored in a specified storage.
`optuna.TrialPruned`	Exception for pruned trials.

optuna.cli¶

optuna
    [--version]
    [-v | -q]
    [--log-file LOG_FILE]
    [--debug]
    [--storage STORAGE]

--version¶: show program’s version number and exit

-v, --verbose¶: Increase verbosity of output. Can be repeated.

-q, --quiet¶: Suppress output except warnings and errors.

--log-file <LOG_FILE>¶: Specify a file to log output. Disabled by default.

--debug¶: Show tracebacks on errors.

--storage <STORAGE>¶: DB URL. (e.g. sqlite:///example.db)

create-study¶

Create a new study.

optuna create-study
    [--study-name STUDY_NAME]
    [--direction {minimize,maximize}]
    [--skip-if-exists]

--study-name <STUDY_NAME>¶: A human-readable name of a study to distinguish it from others.

--direction <DIRECTION>¶: Set direction of optimization to a new study. Set ‘minimize’ for minimization and ‘maximize’ for maximization.

--skip-if-exists¶: If specified, the creation of the study is skipped without any error when the study name is duplicated.

This command is provided by the optuna plugin.

dashboard¶

Launch web dashboard (beta).

optuna dashboard
    [--study STUDY]
    [--study-name STUDY_NAME]
    [--out OUT]
    [--allow-websocket-origin BOKEH_ALLOW_WEBSOCKET_ORIGINS]

--study <STUDY>¶: This argument is deprecated. Use –study-name instead.

--study-name <STUDY_NAME>¶: A human-readable name of a study to distinguish it from others.

--out <OUT>, -o <OUT>¶: Output HTML file path. If it is not given, a HTTP server starts and the dashboard is served.

--allow-websocket-origin <BOKEH_ALLOW_WEBSOCKET_ORIGINS>¶: Allow websocket access from the specified host(s).Internally, it is used as the value of bokeh’s –allow-websocket-origin option. Please refer to https://bokeh.pydata.org/en/latest/docs/reference/command/subcommands/serve.html for more details.

This command is provided by the optuna plugin.

delete-study¶

Delete a specified study.

optuna delete-study [--study-name STUDY_NAME]

--study-name <STUDY_NAME>¶: A human-readable name of a study to distinguish it from others.

This command is provided by the optuna plugin.

storage upgrade¶

Upgrade the schema of a storage.

optuna storage upgrade

This command is provided by the optuna plugin.

studies¶

Show a list of studies.

optuna studies
    [-f {csv,json,table,value,yaml}]
    [-c COLUMN]
    [--quote {all,minimal,none,nonnumeric}]
    [--noindent]
    [--max-width <integer>]
    [--fit-width]
    [--print-empty]
    [--sort-column SORT_COLUMN]

-f <FORMATTER>, --format <FORMATTER>¶: the output format, defaults to table

-c COLUMN, --column COLUMN¶: specify the column(s) to include, can be repeated to show multiple columns

--quote <QUOTE_MODE>¶: when to include quotes, defaults to nonnumeric

--noindent¶: whether to disable indenting the JSON

--max-width <integer>¶: Maximum display width, <1 to disable. You can also use the CLIFF_MAX_TERM_WIDTH environment variable, but the parameter takes precedence.

--fit-width¶: Fit the table to the display width. Implied if –max-width greater than 0. Set the environment variable CLIFF_FIT_WIDTH=1 to always enable

--print-empty¶: Print empty table if there is no data to show.

--sort-column SORT_COLUMN¶: specify the column(s) to sort the data (columns specified first have a priority, non-existing columns are ignored), can be repeated

This command is provided by the optuna plugin.

study optimize¶

Start optimization of a study. Deprecated since version 2.0.0.

optuna study optimize
    [--n-trials N_TRIALS]
    [--timeout TIMEOUT]
    [--n-jobs N_JOBS]
    [--study STUDY]
    [--study-name STUDY_NAME]
    file
    method

--n-trials <N_TRIALS>¶: The number of trials. If this argument is not given, as many trials run as possible.

--timeout <TIMEOUT>¶: Stop study after the given number of second(s). If this argument is not given, as many trials run as possible.

--n-jobs <N_JOBS>¶: The number of parallel jobs. If this argument is set to -1, the number is set to CPU counts.

--study <STUDY>¶: This argument is deprecated. Use –study-name instead.

--study-name <STUDY_NAME>¶: A human-readable name of a study to distinguish it from others.

file¶: Python script file where the objective function resides.

method¶: The method name of the objective function.

This command is provided by the optuna plugin.

study set-user-attr¶

Set a user attribute to a study.

optuna study set-user-attr
    [--study STUDY]
    [--study-name STUDY_NAME]
    --key KEY
    --value VALUE

--study <STUDY>¶: This argument is deprecated. Use –study-name instead.

--study-name <STUDY_NAME>¶: A human-readable name of a study to distinguish it from others.

--key <KEY>, -k <KEY>¶: Key of the user attribute.

--value <VALUE>, -v <VALUE>¶: Value to be set.

This command is provided by the optuna plugin.

optuna.distributions¶

`optuna.distributions.UniformDistribution`	A uniform distribution in the linear domain.
`optuna.distributions.LogUniformDistribution`	A uniform distribution in the log domain.
`optuna.distributions.DiscreteUniformDistribution`	A discretized uniform distribution in the linear domain.
`optuna.distributions.IntUniformDistribution`	A uniform distribution on integers.
`optuna.distributions.IntLogUniformDistribution`	A uniform distribution on integers in the log domain.
`optuna.distributions.CategoricalDistribution`	A categorical distribution.
`optuna.distributions.distribution_to_json`	Serialize a distribution to JSON format.
`optuna.distributions.json_to_distribution`	Deserialize a distribution in JSON format.
`optuna.distributions.check_distribution_compatibility`	A function to check compatibility of two distributions.

optuna.exceptions¶

`optuna.exceptions.OptunaError`	Base class for Optuna specific errors.
`optuna.exceptions.TrialPruned`	Exception for pruned trials.
`optuna.exceptions.CLIUsageError`	Exception for CLI.
`optuna.exceptions.StorageInternalError`	Exception for storage operation.
`optuna.exceptions.DuplicatedStudyError`	Exception for a duplicated study name.

optuna.importance¶

`optuna.importance.get_param_importances`	Evaluate parameter importances based on completed trials in the given study.
`optuna.importance.FanovaImportanceEvaluator`	fANOVA parameter importance evaluator.
`optuna.importance.MeanDecreaseImpurityImportanceEvaluator`	Mean Decrease Impurity (MDI) parameter importance evaluator.

optuna.integration¶

`optuna.integration.ChainerPruningExtension`	Chainer extension to prune unpromising trials.
`optuna.integration.ChainerMNStudy`	A wrapper of `Study` to incorporate Optuna with ChainerMN.
`optuna.integration.PyCmaSampler`	A Sampler using cma library as the backend.
`optuna.integration.CmaEsSampler`	Wrapper class of PyCmaSampler for backward compatibility.
`optuna.integration.FastAIPruningCallback`	FastAI callback to prune unpromising trials for fastai.
`optuna.integration.KerasPruningCallback`	Keras callback to prune unpromising trials.
`optuna.integration.LightGBMPruningCallback`	Callback for LightGBM to prune unpromising trials.
`optuna.integration.lightgbm.train`	Wrapper of LightGBM Training API to tune hyperparameters.
`optuna.integration.lightgbm.LightGBMTuner`	Hyperparameter tuner for LightGBM.
`optuna.integration.lightgbm.LightGBMTunerCV`	Hyperparameter tuner for LightGBM with cross-validation.
`optuna.integration.MLflowCallback`	Callback to track Optuna trials with MLflow.
`optuna.integration.MXNetPruningCallback`	MXNet callback to prune unpromising trials.
`optuna.integration.PyTorchIgnitePruningHandler`	PyTorch Ignite handler to prune unpromising trials.
`optuna.integration.PyTorchLightningPruningCallback`	PyTorch Lightning callback to prune unpromising trials.
`optuna.integration.SkoptSampler`	Sampler using Scikit-Optimize as the backend.
`optuna.integration.TensorFlowPruningHook`	TensorFlow SessionRunHook to prune unpromising trials.
`optuna.integration.TFKerasPruningCallback`	tf.keras callback to prune unpromising trials.
`optuna.integration.XGBoostPruningCallback`	Callback for XGBoost to prune unpromising trials.
`optuna.integration.OptunaSearchCV`	Hyperparameter search with cross-validation.
`optuna.integration.AllenNLPExecutor`	AllenNLP extension to use optuna with Jsonnet config file.
`optuna.integration.allennlp.dump_best_config`	Save JSON config file after updating with parameters from the best trial in the study.

optuna.logging¶

`optuna.logging.get_verbosity`	Return the current level for the Optuna’s root logger.
`optuna.logging.set_verbosity`	Set the level for the Optuna’s root logger.
`optuna.logging.disable_default_handler`	Disable the default handler of the Optuna’s root logger.
`optuna.logging.enable_default_handler`	Enable the default handler of the Optuna’s root logger.
`optuna.logging.disable_propagation`	Disable propagation of the library log outputs.
`optuna.logging.enable_propagation`	Enable propagation of the library log outputs.

optuna.multi_objective¶

optuna.multi_objective.samplers¶

`optuna.multi_objective.samplers.BaseMultiObjectiveSampler`	Base class for multi-objective samplers.
`optuna.multi_objective.samplers.NSGAIIMultiObjectiveSampler`	Multi-objective sampler using the NSGA-II algorithm.
`optuna.multi_objective.samplers.RandomMultiObjectiveSampler`	Multi-objective sampler using random sampling.

optuna.multi_objective.study¶

`optuna.multi_objective.study.MultiObjectiveStudy`	A study corresponds to a multi-objective optimization task, i.e., a set of trials.
`optuna.multi_objective.study.create_study`	Create a new `MultiObjectiveStudy`.
`optuna.multi_objective.study.load_study`	Load the existing `MultiObjectiveStudy` that has the specified name.

optuna.multi_objective.trial¶

`optuna.multi_objective.trial.MultiObjectiveTrial`	A trial is a process of evaluating an objective function.
`optuna.multi_objective.trial.FrozenMultiObjectiveTrial`	Status and results of a `MultiObjectiveTrial`.

optuna.multi_objective.visualization¶

Note

visualization module uses plotly to create figures, but JupyterLab cannot render them by default. Please follow this installation guide to show figures in JupyterLab.

optuna.multi_objective.visualization.plot_pareto_front

Plot the pareto front of a study.

optuna.pruners¶

`optuna.pruners.BasePruner`	Base class for pruners.
`optuna.pruners.MedianPruner`	Pruner using the median stopping rule.
`optuna.pruners.NopPruner`	Pruner which never prunes trials.
`optuna.pruners.PercentilePruner`	Pruner to keep the specified percentile of the trials.
`optuna.pruners.SuccessiveHalvingPruner`	Pruner using Asynchronous Successive Halving Algorithm.
`optuna.pruners.HyperbandPruner`	Pruner using Hyperband.
`optuna.pruners.ThresholdPruner`	Pruner to detect outlying metrics of the trials.

optuna.samplers¶

`optuna.samplers.BaseSampler`	Base class for samplers.
`optuna.samplers.GridSampler`	Sampler using grid search.
`optuna.samplers.RandomSampler`	Sampler using random sampling.
`optuna.samplers.TPESampler`	Sampler using TPE (Tree-structured Parzen Estimator) algorithm.
`optuna.samplers.CmaEsSampler`	A Sampler using CMA-ES algorithm.
`optuna.samplers.IntersectionSearchSpace`	A class to calculate the intersection search space of a `BaseStudy`.
`optuna.samplers.intersection_search_space`	Return the intersection search space of the `BaseStudy`.

optuna.storages¶

`optuna.storages.RDBStorage`	Storage class for RDB backend.
`optuna.storages.RedisStorage`	Storage class for Redis backend.

optuna.structs¶

class optuna.structs.TrialState[source]¶

State of a Trial.

RUNNING¶: The Trial is running.

COMPLETE¶: The Trial has been finished without any error.

PRUNED¶: The Trial has been pruned with TrialPruned.

FAIL¶: The Trial has failed due to an uncaught error.

Deprecated since version 1.4.0: This class is deprecated. Please use TrialState instead.

class optuna.structs.StudyDirection[source]¶

Direction of a Study.

NOT_SET¶: Direction has not been set.

MINIMIZE¶: Study minimizes the objective function.

MAXIMIZE¶: Study maximizes the objective function.

Deprecated since version 1.4.0: This class is deprecated. Please use StudyDirection instead.

class optuna.structs.FrozenTrial(number: int, state: TrialState, value: Optional[float], datetime_start: Optional[datetime], datetime_complete: Optional[datetime], params: Dict[str, Any], distributions: Dict[str, BaseDistribution], user_attrs: Dict[str, Any], system_attrs: Dict[str, Any], intermediate_values: Dict[int, float], trial_id: int)[source]¶

Status and results of a Trial.

number¶: Unique and consecutive number of Trial for each Study. Note that this field uses zero-based numbering.

state¶: TrialState of the Trial.

value¶: Objective value of the Trial.

datetime_start¶: Datetime where the Trial started.

datetime_complete¶: Datetime where the Trial finished.

params¶: Dictionary that contains suggested parameters.

user_attrs¶: Dictionary that contains the attributes of the Trial set with optuna.trial.Trial.set_user_attr().

intermediate_values¶: Intermediate objective values set with optuna.trial.Trial.report().

Warning

Deprecated in v1.4.0. This feature will be removed in the future. The removal of this feature is currently scheduled for v3.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

This class was moved to trial. Please use FrozenTrial instead.

property distributions¶: Dictionary that contains the distributions of params.

property duration¶

Return the elapsed time taken to complete the trial.

Returns: The duration.

class optuna.structs.StudySummary(study_name: str, direction: _study_direction.StudyDirection, best_trial: Optional[FrozenTrial], user_attrs: Dict[str, Any], system_attrs: Dict[str, Any], n_trials: int, datetime_start: Optional[datetime], study_id: int)[source]¶

Basic attributes and aggregated results of a Study.

study_name¶: Name of the Study.

direction¶: StudyDirection of the Study.

best_trial¶: FrozenTrial with best objective value in the Study.

user_attrs¶: Dictionary that contains the attributes of the Study set with optuna.study.Study.set_user_attr().

system_attrs¶: Dictionary that contains the attributes of the Study internally set by Optuna.

n_trials¶: The number of trials ran in the Study.

datetime_start¶: Datetime where the Study started.

Warning

Deprecated in v1.4.0. This feature will be removed in the future. The removal of this feature is currently scheduled for v3.0.0, but this schedule is subject to change. See https://github.com/optuna/optuna/releases/tag/v1.4.0.

This class was moved to study. Please use StudySummary instead.

optuna.study¶

`optuna.study.Study`	A study corresponds to an optimization task, i.e., a set of trials.
`optuna.study.create_study`	Create a new `Study`.
`optuna.study.load_study`	Load the existing `Study` that has the specified name.
`optuna.study.delete_study`	Delete a `Study` object.
`optuna.study.get_all_study_summaries`	Get all history of studies stored in a specified storage.
`optuna.study.StudyDirection`	Direction of a `Study`.
`optuna.study.StudySummary`	Basic attributes and aggregated results of a `Study`.

optuna.trial¶

`optuna.trial.Trial`	A trial is a process of evaluating an objective function.
`optuna.trial.FixedTrial`	A trial class which suggests a fixed value for each parameter.
`optuna.trial.FrozenTrial`	Status and results of a `Trial`.
`optuna.trial.FrozenTrial`	Status and results of a `Trial`.
`optuna.trial.TrialState`	State of a `Trial`.

optuna.visualization¶

Note

visualization module uses plotly to create figures, but JupyterLab cannot render them by default. Please follow this installation guide to show figures in JupyterLab.

`optuna.visualization.plot_contour`	Plot the parameter relationship as contour plot in a study.
`optuna.visualization.plot_intermediate_values`	Plot intermediate values of all trials in a study.
`optuna.visualization.plot_optimization_history`	Plot optimization history of all trials in a study.
`optuna.visualization.plot_parallel_coordinate`	Plot the high-dimentional parameter relationships in a study.
`optuna.visualization.plot_param_importances`	Plot hyperparameter importances.
`optuna.visualization.plot_slice`	Plot the parameter relationship as slice plot in a study.
`optuna.visualization.is_available`	Returns whether visualization is available or not.

FAQ¶

Can I use Optuna with X? (where X is your favorite ML library)
How to define objective functions that have own arguments?
Can I use Optuna without remote RDB servers?
How can I save and resume studies?
How to suppress log messages of Optuna?
How to save machine learning models trained in objective functions?
How can I obtain reproducible optimization results?
How are exceptions from trials handled?
How are NaNs returned by trials handled?
What happens when I dynamically alter a search space?
How can I use two GPUs for evaluating two trials simultaneously?
How can I test my objective functions?
How do I avoid running out of memory (OOM) when optimizing studies?

Can I use Optuna with X? (where X is your favorite ML library)¶

Optuna is compatible with most ML libraries, and it’s easy to use Optuna with those. Please refer to examples.

How to define objective functions that have own arguments?¶

There are two ways to realize it.

First, callable classes can be used for that purpose as follows:

import optuna

class Objective(object):
    def __init__(self, min_x, max_x):
        # Hold this implementation specific arguments as the fields of the class.
        self.min_x = min_x
        self.max_x = max_x

    def __call__(self, trial):
        # Calculate an objective value by using the extra arguments.
        x = trial.suggest_uniform('x', self.min_x, self.max_x)
        return (x - 2) ** 2

# Execute an optimization by using an `Objective` instance.
study = optuna.create_study()
study.optimize(Objective(-100, 100), n_trials=100)

Second, you can use lambda or functools.partial for creating functions (closures) that hold extra arguments. Below is an example that uses lambda:

import optuna

# Objective function that takes three arguments.
def objective(trial, min_x, max_x):
    x = trial.suggest_uniform('x', min_x, max_x)
    return (x - 2) ** 2

# Extra arguments.
min_x = -100
max_x = 100

# Execute an optimization by using the above objective function wrapped by `lambda`.
study = optuna.create_study()
study.optimize(lambda trial: objective(trial, min_x, max_x), n_trials=100)

Please also refer to sklearn_addtitional_args.py example.

Can I use Optuna without remote RDB servers?¶

Yes, it’s possible.

In the simplest form, Optuna works with in-memory storage:

study = optuna.create_study()
study.optimize(objective)

If you want to save and resume studies, it’s handy to use SQLite as the local storage:

study = optuna.create_study(study_name='foo_study', storage='sqlite:///example.db')
study.optimize(objective)  # The state of `study` will be persisted to the local SQLite file.

Please see Saving/Resuming Study with RDB Backend for more details.

How can I save and resume studies?¶

There are two ways of persisting studies, which depends if you are using in-memory storage (default) or remote databases (RDB). In-memory studies can be saved and loaded like usual Python objects using pickle or joblib. For example, using joblib:

study = optuna.create_study()
joblib.dump(study, 'study.pkl')

And to resume the study:

study = joblib.load('study.pkl')
print('Best trial until now:')
print(' Value: ', study.best_trial.value)
print(' Params: ')
for key, value in study.best_trial.params.items():
    print(f'    {key}: {value}')

If you are using RDBs, see Saving/Resuming Study with RDB Backend for more details.

How to suppress log messages of Optuna?¶

By default, Optuna shows log messages at the optuna.logging.INFO level. You can change logging levels by using optuna.logging.set_verbosity().

For instance, you can stop showing each trial result as follows:

optuna.logging.set_verbosity(optuna.logging.WARNING)

study = optuna.create_study()
study.optimize(objective)
# Logs like '[I 2018-12-05 11:41:42,324] Finished a trial resulted in value:...' are disabled.

Please refer to optuna.logging for further details.

How to save machine learning models trained in objective functions?¶

Optuna saves hyperparameter values with its corresponding objective value to storage, but it discards intermediate objects such as machine learning models and neural network weights. To save models or weights, please use features of the machine learning library you used.

We recommend saving optuna.trial.Trial.number with a model in order to identify its corresponding trial. For example, you can save SVM models trained in the objective function as follows:

def objective(trial):
    svc_c = trial.suggest_loguniform('svc_c', 1e-10, 1e10)
    clf = sklearn.svm.SVC(C=svc_c)
    clf.fit(X_train, y_train)

    # Save a trained model to a file.
    with open('{}.pickle'.format(trial.number), 'wb') as fout:
        pickle.dump(clf, fout)
    return 1.0 - accuracy_score(y_valid, clf.predict(X_valid))


study = optuna.create_study()
study.optimize(objective, n_trials=100)

# Load the best model.
with open('{}.pickle'.format(study.best_trial.number), 'rb') as fin:
    best_clf = pickle.load(fin)
print(accuracy_score(y_valid, best_clf.predict(X_valid)))

How can I obtain reproducible optimization results?¶

To make the parameters suggested by Optuna reproducible, you can specify a fixed random seed via seed argument of RandomSampler or TPESampler as follows:

sampler = TPESampler(seed=10)  # Make the sampler behave in a deterministic way.
study = optuna.create_study(sampler=sampler)
study.optimize(objective)

However, there are two caveats.

First, when optimizing a study in distributed or parallel mode, there is inherent non-determinism. Thus it is very difficult to reproduce the same results in such condition. We recommend executing optimization of a study sequentially if you would like to reproduce the result.

Second, if your objective function behaves in a non-deterministic way (i.e., it does not return the same value even if the same parameters were suggested), you cannot reproduce an optimization. To deal with this problem, please set an option (e.g., random seed) to make the behavior deterministic if your optimization target (e.g., an ML library) provides it.

How are exceptions from trials handled?¶

Trials that raise exceptions without catching them will be treated as failures, i.e. with the FAIL status.

By default, all exceptions except TrialPruned raised in objective functions are propagated to the caller of optimize(). In other words, studies are aborted when such exceptions are raised. It might be desirable to continue a study with the remaining trials. To do so, you can specify in optimize() which exception types to catch using the catch argument. Exceptions of these types are caught inside the study and will not propagate further.

You can find the failed trials in log messages.

[W 2018-12-07 16:38:36,889] Setting status of trial#0 as TrialState.FAIL because of \
the following error: ValueError('A sample error in objective.')

You can also find the failed trials by checking the trial states as follows:

study.trials_dataframe()

number	state	value	…	params	system_attrs
0	TrialState.FAIL		…	0	Setting status of trial#0 as TrialState.FAIL because of the following error: ValueError(‘A test error in objective.’)
1	TrialState.COMPLETE	1269	…	1

How are NaNs returned by trials handled?¶

Trials that return NaN (float('nan')) are treated as failures, but they will not abort studies.

Trials which return NaN are shown as follows:

[W 2018-12-07 16:41:59,000] Setting status of trial#2 as TrialState.FAIL because the \
objective function returned nan.

What happens when I dynamically alter a search space?¶

Since parameters search spaces are specified in each call to the suggestion API, e.g. suggest_uniform() and suggest_int(), it is possible to in a single study alter the range by sampling parameters from different search spaces in different trials. The behavior when altered is defined by each sampler individually.

Note

Discussion about the TPE sampler. https://github.com/optuna/optuna/issues/822

How can I use two GPUs for evaluating two trials simultaneously?¶

If your optimization target supports GPU (CUDA) acceleration and you want to specify which GPU is used, the easiest way is to set CUDA_VISIBLE_DEVICES environment variable:

# On a terminal.
#
# Specify to use the first GPU, and run an optimization.
$ export CUDA_VISIBLE_DEVICES=0
$ optuna study optimize foo.py objective --study-name foo --storage sqlite:///example.db

# On another terminal.
#
# Specify to use the second GPU, and run another optimization.
$ export CUDA_VISIBLE_DEVICES=1
$ optuna study optimize bar.py objective --study-name bar --storage sqlite:///example.db

Please refer to CUDA C Programming Guide for further details.

How can I test my objective functions?¶

When you test objective functions, you may prefer fixed parameter values to sampled ones. In that case, you can use FixedTrial, which suggests fixed parameter values based on a given dictionary of parameters. For instance, you can input arbitrary values of \(x\) and \(y\) to the objective function \(x + y\) as follows:

def objective(trial):
    x = trial.suggest_uniform('x', -1.0, 1.0)
    y = trial.suggest_int('y', -5, 5)
    return x + y

objective(FixedTrial({'x': 1.0, 'y': -1}))  # 0.0
objective(FixedTrial({'x': -1.0, 'y': -4}))  # -5.0

Using FixedTrial, you can write unit tests as follows:

# A test function of pytest
def test_objective():
    assert 1.0 == objective(FixedTrial({'x': 1.0, 'y': 0}))
    assert -1.0 == objective(FixedTrial({'x': 0.0, 'y': -1}))
    assert 0.0 == objective(FixedTrial({'x': -1.0, 'y': 1}))

How do I avoid running out of memory (OOM) when optimizing studies?¶

If the memory footprint increases as you run more trials, try to periodically run the garbage collector. Specify gc_after_trial to True when calling optimize() or call gc.collect() inside a callback.

def objective(trial):
    x = trial.suggest_uniform('x', -1.0, 1.0)
    y = trial.suggest_int('y', -5, 5)
    return x + y

study = optuna.create_study()
study.optimize(objective, n_trials=10, gc_after_trial=True)

# `gc_after_trial=True` is more or less identical to the following.
study.optimize(objective, n_trials=10, callbacks=[lambda study, trial: gc.collect()])

There is a performance trade-off for running the garbage collector, which could be non-negligible depending on how fast your objective function otherwise is. Therefore, gc_after_trial is False by default. Note that the above examples are similar to running the garbage collector inside the objective function, except for the fact that gc.collect() is called even when errors, including TrialPruned are raised.

Note

ChainerMNStudy does currently not provide gc_after_trial nor callbacks for optimize(). When using this class, you will have to call the garbage collector inside the objective function.

Optuna: A hyperparameter optimization framework¶

Key Features¶

Basic Concepts¶

Communication¶

Contribution¶

License¶

Reference¶

Installation¶

Tutorial¶

First Optimization¶

Quadratic Function Example¶

Study Object¶

Advanced Configurations¶

Defining Parameter Spaces¶

Branches and Loops¶

Note on the Number of Parameters¶

Arguments for Study.optimize¶

Saving/Resuming Study with RDB Backend¶

New Study¶

Resume Study¶

Experimental History¶

Distributed Optimization¶

Command-Line Interface¶

User Attributes¶

Adding User Attributes to Studies¶

Adding User Attributes to Trials¶

Pruning Unpromising Trials¶

Activating Pruners¶

Integration Modules for Pruning¶

User-Defined Sampler¶

Overview of Sampler¶

An Example: Implementing SimulatedAnnealingSampler¶

API Reference¶

optuna¶

optuna.cli¶

create-study¶

dashboard¶

delete-study¶

storage upgrade¶

studies¶

study optimize¶

study set-user-attr¶

optuna.distributions¶

optuna.exceptions¶

optuna.importance¶

optuna.integration¶

optuna.logging¶

optuna.multi_objective¶

optuna.multi_objective.samplers¶

optuna.multi_objective.study¶

optuna.multi_objective.trial¶

optuna.multi_objective.visualization¶

optuna.pruners¶

optuna.samplers¶

optuna.storages¶

optuna.structs¶

optuna.study¶

optuna.trial¶

optuna.visualization¶

FAQ¶

Indices and tables¶