{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n\n# Pythonic Search Space\n\nFor hyperparameter sampling, Optuna provides the following features:\n\n- :func:`optuna.trial.Trial.suggest_categorical` for categorical parameters\n- :func:`optuna.trial.Trial.suggest_int` for integer parameters\n- :func:`optuna.trial.Trial.suggest_float` for floating point parameters\n\nWith optional arguments of ``step`` and ``log``, we can discretize or take the logarithm of\ninteger and floating point parameters.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import optuna\n\n\ndef objective(trial):\n    # Categorical parameter\n    optimizer = trial.suggest_categorical(\"optimizer\", [\"MomentumSGD\", \"Adam\"])\n\n    # Integer parameter\n    num_layers = trial.suggest_int(\"num_layers\", 1, 3)\n\n    # Integer parameter (log)\n    num_channels = trial.suggest_int(\"num_channels\", 32, 512, log=True)\n\n    # Integer parameter (discretized)\n    num_units = trial.suggest_int(\"num_units\", 10, 100, step=5)\n\n    # Floating point parameter\n    dropout_rate = trial.suggest_float(\"dropout_rate\", 0.0, 1.0)\n\n    # Floating point parameter (log)\n    learning_rate = trial.suggest_float(\"learning_rate\", 1e-5, 1e-2, log=True)\n\n    # Floating point parameter (discretized)\n    drop_path_rate = trial.suggest_float(\"drop_path_rate\", 0.0, 1.0, step=0.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Defining Parameter Spaces\n\nIn Optuna, we define search spaces using familiar Python syntax including conditionals and loops.\n\nAlso, you can use branches or loops depending on the parameter values.\n\nFor more various use, see `examples <https://github.com/optuna/optuna/tree/master/examples>`_.\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "- Branches:\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "import sklearn.ensemble\nimport sklearn.svm\n\n\ndef objective(trial):\n    classifier_name = trial.suggest_categorical(\"classifier\", [\"SVC\", \"RandomForest\"])\n    if classifier_name == \"SVC\":\n        svc_c = trial.suggest_float(\"svc_c\", 1e-10, 1e10, log=True)\n        classifier_obj = sklearn.svm.SVC(C=svc_c)\n    else:\n        rf_max_depth = trial.suggest_int(\"rf_max_depth\", 2, 32, log=True)\n        classifier_obj = sklearn.ensemble.RandomForestClassifier(max_depth=rf_max_depth)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "- Loops:\n\n.. code-block:: python\n\n    import torch\n    import torch.nn as nn\n\n\n    def create_model(trial, in_size):\n        n_layers = trial.suggest_int(\"n_layers\", 1, 3)\n\n        layers = []\n        for i in range(n_layers):\n            n_units = trial.suggest_int(\"n_units_l{}\".format(i), 4, 128, log=True)\n            layers.append(nn.Linear(in_size, n_units))\n            layers.append(nn.ReLU())\n            in_size = n_units\n        layers.append(nn.Linear(in_size, 10))\n\n        return nn.Sequential(*layers)\n\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Note on the Number of Parameters\n\nThe difficulty of optimization increases roughly exponentially with regard to the number of parameters. That is, the number of necessary trials increases exponentially when you increase the number of parameters, so it is recommended to not add unimportant parameters.\n\n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}