Note
Click here to download the full example code
Lightweight, versatile, and platform agnostic architecture¶
Optuna is entirely written in Python and has few dependencies. This means that we can quickly move to the real example once you get interested in Optuna.
Quadratic Function Example¶
Usually, Optuna is used to optimize hyperparameters, but as an example, let’s optimize a simple quadratic function: \((x - 2)^2\).
First of all, import optuna.
import optuna
In optuna, conventionally functions to be optimized are named objective.
def objective(trial):
x = trial.suggest_float("x", -10, 10)
return (x - 2) ** 2
This function returns the value of \((x - 2)^2\). Our goal is to find the value of x
that minimizes the output of the objective function. This is the “optimization.”
During the optimization, Optuna repeatedly calls and evaluates the objective function with
different values of x.
A Trial object corresponds to a single execution of the objective
function and is internally instantiated upon each invocation of the function.
The suggest APIs (for example, suggest_float()) are called
inside the objective function to obtain parameters for a trial.
suggest_float() selects parameters uniformly within the range
provided. In our example, from \(-10\) to \(10\).
To start the optimization, we create a study object and pass the objective function to method
optimize() as follows.
study = optuna.create_study()
study.optimize(objective, n_trials=100)
You can get the best parameter as follows.
best_params = study.best_params
found_x = best_params["x"]
print("Found x: {}, (x - 2)^2: {}".format(found_x, (found_x - 2) ** 2))
Out:
Found x: 2.0016016629797964, (x - 2)^2: 2.5653243008501516e-06
We can see that the x value found by Optuna is close to the optimal value of 2.
Note
When used to search for hyperparameters in machine learning, usually the objective function would return the loss or accuracy of the model.
Study Object¶
Let us clarify the terminology in Optuna as follows:
Trial: A single call of the objective function
Study: An optimization session, which is a set of trials
Parameter: A variable whose value is to be optimized, such as
xin the above example
In Optuna, we use the study object to manage optimization.
Method create_study() returns a study object.
A study object has useful properties for analyzing the optimization outcome.
To get the dictionary of parameter name and parameter values:
study.best_params
Out:
{'x': 2.0016016629797964}
To get the best observed value of the objective function:
study.best_value
Out:
2.5653243008501516e-06
To get the best trial:
study.best_trial
Out:
FrozenTrial(number=82, value=2.5653243008501516e-06, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 578971), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 582270), params={'x': 2.0016016629797964}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=82, state=TrialState.COMPLETE)
To get all trials:
study.trials
Out:
[FrozenTrial(number=0, value=65.70855554403302, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 315660), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 315855), params={'x': -6.106081392635594}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=0, state=TrialState.COMPLETE), FrozenTrial(number=1, value=32.90935193400542, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 316153), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 316328), params={'x': -3.736667319446494}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=1, state=TrialState.COMPLETE), FrozenTrial(number=2, value=62.42959099793248, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 316636), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 316806), params={'x': -5.901239839286774}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=2, state=TrialState.COMPLETE), FrozenTrial(number=3, value=35.26674392921999, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 317095), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 317257), params={'x': -3.938580969324237}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=3, state=TrialState.COMPLETE), FrozenTrial(number=4, value=36.19649272227829, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 317549), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 317717), params={'x': -4.016352110895629}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=4, state=TrialState.COMPLETE), FrozenTrial(number=5, value=14.438419464348696, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 318009), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 318179), params={'x': 5.799792029091684}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=5, state=TrialState.COMPLETE), FrozenTrial(number=6, value=62.88312159010636, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 318458), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 318629), params={'x': -5.929887867435855}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=6, state=TrialState.COMPLETE), FrozenTrial(number=7, value=29.196116929196613, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 318913), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 319084), params={'x': 7.403343125250942}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=7, state=TrialState.COMPLETE), FrozenTrial(number=8, value=83.66513766540027, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 319375), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 319551), params={'x': -7.146864909104118}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=8, state=TrialState.COMPLETE), FrozenTrial(number=9, value=135.1044205963803, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 319823), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 320009), params={'x': -9.623442717043014}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=9, state=TrialState.COMPLETE), FrozenTrial(number=10, value=20.08410990921703, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 320281), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 324526), params={'x': 6.4815298625823115}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=10, state=TrialState.COMPLETE), FrozenTrial(number=11, value=23.363682844721218, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 325008), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 331884), params={'x': 6.833599367419813}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=11, state=TrialState.COMPLETE), FrozenTrial(number=12, value=0.9601744059033999, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 332220), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 335235), params={'x': 2.9798848942112537}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=12, state=TrialState.COMPLETE), FrozenTrial(number=13, value=0.03733581771348687, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 335598), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 338605), params={'x': 2.1932247854533338}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=13, state=TrialState.COMPLETE), FrozenTrial(number=14, value=0.038607703475723565, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 338890), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 341954), params={'x': 2.1964884308953674}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=14, state=TrialState.COMPLETE), FrozenTrial(number=15, value=1.9167591525847496, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 342319), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 345519), params={'x': 0.6155292879281449}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=15, state=TrialState.COMPLETE), FrozenTrial(number=16, value=0.7851845007983917, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 345840), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 349152), params={'x': 2.8861063710404027}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=16, state=TrialState.COMPLETE), FrozenTrial(number=17, value=4.22789567026869, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 349460), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 352510), params={'x': -0.05618473641564847}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=17, state=TrialState.COMPLETE), FrozenTrial(number=18, value=3.3390243271430036, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 352807), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 355770), params={'x': 0.17270026346441902}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=18, state=TrialState.COMPLETE), FrozenTrial(number=19, value=0.4879514579084581, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 356086), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 359112), params={'x': 2.698535223097918}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=19, state=TrialState.COMPLETE), FrozenTrial(number=20, value=6.323711245965134, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 359473), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 362513), params={'x': 4.514699036856127}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=20, state=TrialState.COMPLETE), FrozenTrial(number=21, value=61.501786318667264, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 362779), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 365848), params={'x': 9.84230746137049}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=21, state=TrialState.COMPLETE), FrozenTrial(number=22, value=0.0055090197591712835, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 366168), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 369197), params={'x': 1.9257772288366213}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=22, state=TrialState.COMPLETE), FrozenTrial(number=23, value=0.5907873495665922, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 369507), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 372403), params={'x': 1.2313730751745733}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=23, state=TrialState.COMPLETE), FrozenTrial(number=24, value=13.693817917621196, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 372766), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 375898), params={'x': -1.70051589884724}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=24, state=TrialState.COMPLETE), FrozenTrial(number=25, value=11.77372354635492, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 376266), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 379923), params={'x': -1.4312859901726234}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=25, state=TrialState.COMPLETE), FrozenTrial(number=26, value=6.121342139891911, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 380233), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 383494), params={'x': 4.474134624447892}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=26, state=TrialState.COMPLETE), FrozenTrial(number=27, value=57.51620610037797, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 383804), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 386754), params={'x': 9.583943967381218}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=27, state=TrialState.COMPLETE), FrozenTrial(number=28, value=12.798220271228756, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 387026), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 390398), params={'x': -1.577460030696186}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=28, state=TrialState.COMPLETE), FrozenTrial(number=29, value=0.06961954817646306, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 390748), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 393826), params={'x': 1.736144834849755}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=29, state=TrialState.COMPLETE), FrozenTrial(number=30, value=6.628065116324949, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 394096), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 397153), params={'x': 4.574502887224046}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=30, state=TrialState.COMPLETE), FrozenTrial(number=31, value=0.01427966190174364, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 397479), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 400794), params={'x': 1.8805024606874952}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=31, state=TrialState.COMPLETE), FrozenTrial(number=32, value=0.024296654346231916, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 401100), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 404105), params={'x': 2.155873841122338}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=32, state=TrialState.COMPLETE), FrozenTrial(number=33, value=6.50219201525846, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 404375), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 407483), params={'x': -0.5499396101199063}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=33, state=TrialState.COMPLETE), FrozenTrial(number=34, value=2.208619639790776, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 407754), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 410682), params={'x': 3.486142536835137}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=34, state=TrialState.COMPLETE), FrozenTrial(number=35, value=26.364852635967782, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 410987), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 414346), params={'x': -3.134671619097737}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=35, state=TrialState.COMPLETE), FrozenTrial(number=36, value=0.7864190897987554, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 414653), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 417980), params={'x': 1.1131972655664872}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=36, state=TrialState.COMPLETE), FrozenTrial(number=37, value=10.724929576098349, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 418307), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 421598), params={'x': 5.274893826690928}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=37, state=TrialState.COMPLETE), FrozenTrial(number=38, value=2.8939421479135667, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 421929), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 425026), params={'x': 3.7011590601450433}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=38, state=TrialState.COMPLETE), FrozenTrial(number=39, value=31.98847025052037, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 425332), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 428506), params={'x': 7.655835062174318}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=39, state=TrialState.COMPLETE), FrozenTrial(number=40, value=0.03497900308975568, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 428848), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 431912), params={'x': 1.8129732556831626}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=40, state=TrialState.COMPLETE), FrozenTrial(number=41, value=0.24347948880688178, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 432220), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 435217), params={'x': 1.5065635919321703}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=41, state=TrialState.COMPLETE), FrozenTrial(number=42, value=7.4519216113383075, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 435523), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 438782), params={'x': -0.7298208020561181}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=42, state=TrialState.COMPLETE), FrozenTrial(number=43, value=0.02605669904227884, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 439104), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 442047), params={'x': 2.1614208754847986}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=43, state=TrialState.COMPLETE), FrozenTrial(number=44, value=2.5357580135856503, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 442356), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 445654), params={'x': 3.5924063594402185}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=44, state=TrialState.COMPLETE), FrozenTrial(number=45, value=24.115267653016936, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 445964), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 449050), params={'x': -2.910729849321477}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=45, state=TrialState.COMPLETE), FrozenTrial(number=46, value=0.9577824269882834, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 449357), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 452371), params={'x': 1.0213364076515958}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=46, state=TrialState.COMPLETE), FrozenTrial(number=47, value=0.01975075340287809, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 452682), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 455764), params={'x': 2.1405373736871374}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=47, state=TrialState.COMPLETE), FrozenTrial(number=48, value=0.2198871674088895, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 456073), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 459160), params={'x': 2.468921280609965}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=48, state=TrialState.COMPLETE), FrozenTrial(number=49, value=15.899978030779515, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 459502), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 462773), params={'x': 5.987477652699701}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=49, state=TrialState.COMPLETE), FrozenTrial(number=50, value=2.462797941359173, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 463101), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 466379), params={'x': 0.43066958821312173}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=50, state=TrialState.COMPLETE), FrozenTrial(number=51, value=0.019721313014483564, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 466708), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 470198), params={'x': 1.8595674075775728}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=51, state=TrialState.COMPLETE), FrozenTrial(number=52, value=3.527146223068448, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 470512), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 473722), params={'x': 3.8780698131508444}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=52, state=TrialState.COMPLETE), FrozenTrial(number=53, value=0.8533350737439601, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 474036), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 477360), params={'x': 2.923761372727806}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=53, state=TrialState.COMPLETE), FrozenTrial(number=54, value=5.774828492903198, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 477708), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 480978), params={'x': -0.40308728366307944}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=54, state=TrialState.COMPLETE), FrozenTrial(number=55, value=10.13477900203025, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 481295), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 484708), params={'x': 5.183516766412618}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=55, state=TrialState.COMPLETE), FrozenTrial(number=56, value=0.024957755518050746, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 485045), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 488487), params={'x': 2.157980237745266}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=56, state=TrialState.COMPLETE), FrozenTrial(number=57, value=2.0672822294204587, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 488794), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 492230), params={'x': 0.562195343789547}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=57, state=TrialState.COMPLETE), FrozenTrial(number=58, value=1.4280852286732828, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 492569), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 495787), params={'x': 3.195025200016001}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=58, state=TrialState.COMPLETE), FrozenTrial(number=59, value=9.547780256513228, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 496105), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 499387), params={'x': -1.089948261138563}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=59, state=TrialState.COMPLETE), FrozenTrial(number=60, value=3.4946703190260977, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 499732), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 503047), params={'x': 0.13059626644587619}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=60, state=TrialState.COMPLETE), FrozenTrial(number=61, value=0.08182892103647996, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 503378), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 506684), params={'x': 2.286057548469674}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=61, state=TrialState.COMPLETE), FrozenTrial(number=62, value=0.003613552068857152, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 507023), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 510311), params={'x': 1.9398871721771704}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=62, state=TrialState.COMPLETE), FrozenTrial(number=63, value=1.0590064038990625, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 510617), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 513707), params={'x': 0.9709196319533336}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=63, state=TrialState.COMPLETE), FrozenTrial(number=64, value=4.470432960928841, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 514029), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 517047), params={'x': 4.114339840453479}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=64, state=TrialState.COMPLETE), FrozenTrial(number=65, value=0.04933045337615848, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 517317), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 520732), params={'x': 1.7778953999212117}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=65, state=TrialState.COMPLETE), FrozenTrial(number=66, value=0.6162556895378192, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 521074), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 524404), params={'x': 2.7850195472329458}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=66, state=TrialState.COMPLETE), FrozenTrial(number=67, value=0.12349880032670632, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 524725), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 527793), params={'x': 1.6485760390543833}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=67, state=TrialState.COMPLETE), FrozenTrial(number=68, value=9.173086340933, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 528161), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 531554), params={'x': 5.028710342857666}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=68, state=TrialState.COMPLETE), FrozenTrial(number=69, value=18.657133382029038, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 531893), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 535209), params={'x': -2.319390394723431}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=69, state=TrialState.COMPLETE), FrozenTrial(number=70, value=1.4174220460030702, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 535527), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 538848), params={'x': 3.19055535192744}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=70, state=TrialState.COMPLETE), FrozenTrial(number=71, value=0.04421822793454833, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 539130), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 542113), params={'x': 2.2102813066693003}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=71, state=TrialState.COMPLETE), FrozenTrial(number=72, value=1.4847368257715419, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 542416), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 545641), params={'x': 0.781502225783099}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=72, state=TrialState.COMPLETE), FrozenTrial(number=73, value=0.012141065420648214, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 545962), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 549316), params={'x': 2.1101865028968985}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=73, state=TrialState.COMPLETE), FrozenTrial(number=74, value=4.006248118666752, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 549638), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 552872), params={'x': -0.0015614201584601695}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=74, state=TrialState.COMPLETE), FrozenTrial(number=75, value=4.923276150112382, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 553228), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 556521), params={'x': 4.218845679652459}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=75, state=TrialState.COMPLETE), FrozenTrial(number=76, value=0.4148332959572124, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 556842), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 560076), params={'x': 1.3559244640904202}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=76, state=TrialState.COMPLETE), FrozenTrial(number=77, value=0.341456989350815, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 560400), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 563764), params={'x': 2.5843432119489496}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=77, state=TrialState.COMPLETE), FrozenTrial(number=78, value=115.41183100189464, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 564121), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 567533), params={'x': -8.742989853941715}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=78, state=TrialState.COMPLETE), FrozenTrial(number=79, value=1.628073522123934, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 567877), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 571313), params={'x': 3.2759598434605746}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=79, state=TrialState.COMPLETE), FrozenTrial(number=80, value=0.469588469389763, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 571657), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 574978), params={'x': 1.314734745233816}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=80, state=TrialState.COMPLETE), FrozenTrial(number=81, value=4.1708287942184535e-05, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 575315), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 578621), params={'x': 1.993541804590895}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=81, state=TrialState.COMPLETE), FrozenTrial(number=82, value=2.5653243008501516e-06, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 578971), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 582270), params={'x': 2.0016016629797964}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=82, state=TrialState.COMPLETE), FrozenTrial(number=83, value=0.019710678470537823, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 582600), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 586062), params={'x': 1.8596052762012125}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=83, state=TrialState.COMPLETE), FrozenTrial(number=84, value=2.431242925099394, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 586410), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 589876), params={'x': 0.44075565574237396}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=84, state=TrialState.COMPLETE), FrozenTrial(number=85, value=5.072897465015439, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 590229), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 593719), params={'x': -0.2523093626354793}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=85, state=TrialState.COMPLETE), FrozenTrial(number=86, value=0.6355150692538237, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 594071), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 597529), params={'x': 2.7971919902092743}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=86, state=TrialState.COMPLETE), FrozenTrial(number=87, value=0.01026348881115298, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 597880), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 601270), params={'x': 1.8986911217555293}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=87, state=TrialState.COMPLETE), FrozenTrial(number=88, value=0.15848189086878664, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 601598), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 604873), params={'x': 1.60190215917593}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=88, state=TrialState.COMPLETE), FrozenTrial(number=89, value=1.7195366790397328, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 605205), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 608589), params={'x': 0.6886889464967769}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=89, state=TrialState.COMPLETE), FrozenTrial(number=90, value=2.713215355560109, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 608941), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 612259), params={'x': 3.6471840685121104}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=90, state=TrialState.COMPLETE), FrozenTrial(number=91, value=0.013417366774289006, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 612553), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 615707), params={'x': 1.884166642221297}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=91, state=TrialState.COMPLETE), FrozenTrial(number=92, value=0.06398235724413853, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 616060), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 619132), params={'x': 1.7470526591479196}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=92, state=TrialState.COMPLETE), FrozenTrial(number=93, value=0.8117255926437386, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 619519), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 622556), params={'x': 1.0990418474514265}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=93, state=TrialState.COMPLETE), FrozenTrial(number=94, value=0.3261699318780959, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 622882), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 626289), params={'x': 2.571112888909098}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=94, state=TrialState.COMPLETE), FrozenTrial(number=95, value=1.2586433450589471, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 626585), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 629748), params={'x': 3.1218927511393177}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=95, state=TrialState.COMPLETE), FrozenTrial(number=96, value=8.065045189776898, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 630057), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 633441), params={'x': -0.8399023204640153}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=96, state=TrialState.COMPLETE), FrozenTrial(number=97, value=0.01752206717946293, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 633715), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 636834), params={'x': 1.8676290546250314}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=97, state=TrialState.COMPLETE), FrozenTrial(number=98, value=2.8682836088574164, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 637145), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 640339), params={'x': 0.3063992179803954}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=98, state=TrialState.COMPLETE), FrozenTrial(number=99, value=7.429115915052298, datetime_start=datetime.datetime(2020, 12, 4, 4, 8, 3, 640670), datetime_complete=datetime.datetime(2020, 12, 4, 4, 8, 3, 643906), params={'x': 4.725640459608035}, distributions={'x': UniformDistribution(high=10, low=-10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=99, state=TrialState.COMPLETE)]
To get the number of trials:
len(study.trials)
Out:
100
By executing optimize() again, we can continue the optimization.
study.optimize(objective, n_trials=100)
To get the updated number of trials:
len(study.trials)
Out:
200
As the objective function is so easy that the last 100 trials don’t improve the result. However, we can check the result again:
best_params = study.best_params
found_x = best_params["x"]
print("Found x: {}, (x - 2)^2: {}".format(found_x, (found_x - 2) ** 2))
Out:
Found x: 2.0016016629797964, (x - 2)^2: 2.5653243008501516e-06
Total running time of the script: ( 0 minutes 0.733 seconds)