{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "\n\n# Efficient Optimization Algorithms\n\nOptuna enables efficient hyperparameter optimization by\nadopting state-of-the-art algorithms for sampling hyperparameters and\npruning efficiently unpromising trials.\n\n## Sampling Algorithms\n\nSamplers basically continually narrow down the search space using the records of suggested parameter values and evaluated objective values,\nleading to an optimal search space which giving off parameters leading to better objective values.\nMore detailed explanation of how samplers suggest parameters is in :class:`optuna.samplers.BaseSampler`.\n\nOptuna provides the following sampling algorithms:\n\n- Tree-structured Parzen Estimator algorithm implemented in :class:`optuna.samplers.TPESampler`\n\n- CMA-ES based algorithm implemented in :class:`optuna.samplers.CmaEsSampler`\n\n- Grid Search implemented in :class:`optuna.samplers.GridSampler`\n\n- Random Search implemented in :class:`optuna.samplers.RandomSampler`\n\nThe default sampler is :class:`optuna.samplers.TPESampler`.\n\n## Switching Samplers\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import optuna" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "By default, Optuna uses :class:`~optuna.samplers.TPESampler` as follows.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "study = optuna.create_study()\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If you want to use different samplers for example :class:`~optuna.samplers.RandomSampler`\nand :class:`~optuna.samplers.CmaEsSampler`,\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "study = optuna.create_study(sampler=optuna.samplers.RandomSampler())\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")\n\nstudy = optuna.create_study(sampler=optuna.samplers.CmaEsSampler())\nprint(f\"Sampler is {study.sampler.__class__.__name__}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Pruning Algorithms\n\n``Pruners`` automatically stop unpromising trials at the early stages of the training (a.k.a., automated early-stopping).\n\nOptuna provides the following pruning algorithms:\n\n- Asynchronous Successive Halving algorithm implemted in :class:`optuna.pruners.SuccessiveHalvingPruner`\n\n- Hyperband algorithm implemented in :class:`optuna.pruners.HyperbandPruner`\n\n- Median pruning algorithm implemented in :class:`optuna.pruners.MedianPruner`\n\n- Threshold pruning algorithm implemented in :class:`optuna.pruners.ThresholdPruner`\n\nWe use :class:`optuna.pruners.MedianPruner` in most examples,\nthough basically it is outperformed by :class:`optuna.pruners.SuccessiveHalvingPruner` and\n:class:`optuna.pruners.HyperbandPruner` as in `this benchmark result `_.\n\n\n## Activating Pruners\nTo turn on the pruning feature, you need to call :func:`~optuna.trial.Trial.report` and :func:`~optuna.trial.Trial.should_prune` after each step of the iterative training.\n:func:`~optuna.trial.Trial.report` periodically monitors the intermediate objective values.\n:func:`~optuna.trial.Trial.should_prune` decides termination of the trial that does not meet a predefined condition.\n\nWe would recommend using integration modules for major machine learning frameworks.\nExclusive list is :mod:`optuna.integration` and usecases are available in `optuna/examples `_.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "import logging\nimport sys\n\nimport sklearn.datasets\nimport sklearn.linear_model\nimport sklearn.model_selection\n\n\ndef objective(trial):\n iris = sklearn.datasets.load_iris()\n classes = list(set(iris.target))\n train_x, valid_x, train_y, valid_y = sklearn.model_selection.train_test_split(\n iris.data, iris.target, test_size=0.25, random_state=0\n )\n\n alpha = trial.suggest_loguniform(\"alpha\", 1e-5, 1e-1)\n clf = sklearn.linear_model.SGDClassifier(alpha=alpha)\n\n for step in range(100):\n clf.partial_fit(train_x, train_y, classes=classes)\n\n # Report intermediate objective value.\n intermediate_value = 1.0 - clf.score(valid_x, valid_y)\n trial.report(intermediate_value, step)\n\n # Handle pruning based on the intermediate value.\n if trial.should_prune():\n raise optuna.TrialPruned()\n\n return 1.0 - clf.score(valid_x, valid_y)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Set up the median stopping rule as the pruning condition.\n\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false }, "outputs": [], "source": [ "# Add stream handler of stdout to show the messages\noptuna.logging.get_logger(\"optuna\").addHandler(logging.StreamHandler(sys.stdout))\nstudy = optuna.create_study(pruner=optuna.pruners.MedianPruner())\nstudy.optimize(objective, n_trials=20)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, several trials were pruned (stopped) before they finished all of the iterations.\nThe format of message is ``\"Trial pruned.\"``.\n\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Which Sampler and Pruner Should be Used?\n\nFrom the benchmark results which are available at `optuna/optuna - wiki \"Benchmarks with Kurobako\" `_, at least for not deep learning tasks, we would say that\n\n* For :class:`optuna.samplers.RandomSampler`, :class:`optuna.pruners.MedianPruner` is the best.\n* For :class:`optuna.samplers.TPESampler`, :class:`optuna.pruners.Hyperband` is the best.\n\nHowever, note that the benchmark is not deep learning.\nFor deep learning tasks,\nconsult the below table from `Ozaki et al, Hyperparameter Optimization Methods: Overview and Characteristics, in IEICE Trans, Vol.J103-D No.9 pp.615-631, 2020 `_,\n\n+---------------------------+-----------------------------------------+---------------------------------------------------------------+\n| Parallel Compute Resource | Categorical/Conditional Hyperparameters | Recommended Algorithms |\n+===========================+=========================================+===============================================================+\n| Limited | No | TPE. GP-EI if search space is low-dimensional and continuous. |\n+ +-----------------------------------------+---------------------------------------------------------------+\n| | Yes | TPE. GP-EI if search space is low-dimensional and continuous |\n+---------------------------+-----------------------------------------+---------------------------------------------------------------+\n| Sufficient | No | CMA-ES, Random Search |\n+ +-----------------------------------------+---------------------------------------------------------------+\n| | Yes | Random Search or Genetic Algorithm |\n+---------------------------+-----------------------------------------+---------------------------------------------------------------+\n\n\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Integration Modules for Pruning\nTo implement pruning mechanism in much simpler forms, Optuna provides integration modules for the following libraries.\n\nFor the complete list of Optuna's integration modules, see :mod:`optuna.integration`.\n\nFor example, :class:`~optuna.integration.XGBoostPruningCallback` introduces pruning without directly changing the logic of training iteration.\n(See also `example `_ for the entire script.)\n\n.. code-block:: python\n\n pruning_callback = optuna.integration.XGBoostPruningCallback(trial, 'validation-error')\n bst = xgb.train(param, dtrain, evals=[(dvalid, 'validation')], callbacks=[pruning_callback])\n\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.6" } }, "nbformat": 4, "nbformat_minor": 0 }