{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "50f0c7bc",
   "metadata": {},
   "source": [
    "# Switching the classification metric with `partial_fit`\n",
    "\n",
    "Brush uses the `scorer` to evaluate programs: it drives selection, survival,\n",
    "the archive, and the final model choice. For classification, the available\n",
    "scorers are:\n",
    "\n",
    "| scorer | binary | multiclass |\n",
    "|---|---|---|\n",
    "| `\"log\"` / `\"multi_log\"` | log loss | multinomial log loss |\n",
    "| `\"accuracy\"` | accuracy | accuracy |\n",
    "| `\"balanced_accuracy\"` | balanced accuracy | balanced accuracy |\n",
    "| `\"precision\"` | precision (threshold 0.5) | macro precision |\n",
    "| `\"recall\"` | recall (threshold 0.5) | macro recall |\n",
    "| `\"roc_auc\"` | AUROC | macro one-vs-rest AUROC |\n",
    "| `\"average_precision_score\"` | average precision (AUPRC) | macro one-vs-rest average precision |\n",
    "\n",
    "The scorer is **not** used to fit parameters. Weights are always optimized with\n",
    "the log loss, and split thresholds with the gini impurity. This keeps parameter\n",
    "fitting smooth and well-behaved while letting you pick whichever metric you care\n",
    "about for model selection.\n",
    "\n",
    "Because the scorer is just an estimator attribute, you can change it between\n",
    "calls to `partial_fit`. This notebook:\n",
    "\n",
    "1. Fits a model on an imbalanced binary problem using AUROC.\n",
    "2. Locks every internal node of the best program, leaving only the leaves and\n",
    "   the weights free to change.\n",
    "3. Switches the scorer to average precision and calls `partial_fit`.\n",
    "4. Compares all metrics before and after the switch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "37c250d0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T19:12:04.638378Z",
     "iopub.status.busy": "2026-09-22T19:12:04.638227Z",
     "iopub.status.idle": "2026-09-22T19:12:05.855283Z",
     "shell.execute_reply": "2026-09-22T19:12:05.854853Z"
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import graphviz\n",
    "\n",
    "from sklearn.datasets import make_classification\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import (log_loss, accuracy_score, balanced_accuracy_score,\n",
    "                             precision_score, recall_score, roc_auc_score,\n",
    "                             average_precision_score)\n",
    "\n",
    "from pybrush import BrushClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db851330",
   "metadata": {},
   "source": [
    "## 1. An imbalanced binary problem\n",
    "\n",
    "AUROC and average precision disagree the most when positives are rare: AUROC\n",
    "rewards ranking negatives correctly, while average precision focuses on how\n",
    "clean the top of the ranking is."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c6262431",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T19:12:05.856401Z",
     "iopub.status.busy": "2026-09-22T19:12:05.856312Z",
     "iopub.status.idle": "2026-09-22T19:12:05.865082Z",
     "shell.execute_reply": "2026-09-22T19:12:05.864748Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train prevalence: 0.109\n",
      "test prevalence:  0.107\n"
     ]
    }
   ],
   "source": [
    "X, y = make_classification(\n",
    "    n_samples=1000, n_features=6, n_informative=4, n_redundant=1,\n",
    "    weights=[0.9, 0.1], class_sep=0.8, flip_y=0.02, random_state=42,\n",
    ")\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=0.3, stratify=y, random_state=42)\n",
    "\n",
    "print('train prevalence:', y_train.mean().round(3))\n",
    "print('test prevalence: ', y_test.mean().round(3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ed5afaf0",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T19:12:05.865913Z",
     "iopub.status.busy": "2026-09-22T19:12:05.865867Z",
     "iopub.status.idle": "2026-09-22T19:12:05.867551Z",
     "shell.execute_reply": "2026-09-22T19:12:05.867314Z"
    }
   },
   "outputs": [],
   "source": [
    "def all_metrics(est, X, y):\n",
    "    proba = est.predict_proba(X)[:, 1]\n",
    "    pred = est.predict(X)\n",
    "    return {\n",
    "        'log_loss':          log_loss(y, proba),\n",
    "        'accuracy':          accuracy_score(y, pred),\n",
    "        'balanced_accuracy': balanced_accuracy_score(y, pred),\n",
    "        'precision':         precision_score(y, pred, zero_division=0),\n",
    "        'recall':            recall_score(y, pred, zero_division=0),\n",
    "        'roc_auc':           roc_auc_score(y, proba),\n",
    "        'average_precision': average_precision_score(y, proba),\n",
    "    }\n",
    "\n",
    "def report(est, label):\n",
    "    return pd.DataFrame({\n",
    "        (label, 'train'): all_metrics(est, X_train, y_train),\n",
    "        (label, 'test'):  all_metrics(est, X_test, y_test),\n",
    "    })"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c072f652",
   "metadata": {},
   "source": [
    "## 2. Fit using AUROC as the scorer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bfa036f8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T19:12:05.868283Z",
     "iopub.status.busy": "2026-09-22T19:12:05.868241Z",
     "iopub.status.idle": "2026-09-22T19:12:14.024733Z",
     "shell.execute_reply": "2026-09-22T19:12:14.024331Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scorer: roc_auc\n",
      "internal scorer value (train): 0.9027\n",
      "model: Logistic(Add(0.98,Add(-1.05*Mul(-1.32*x_4,-1.01*x_2),x_0)))\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">roc_auc</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>train</th>\n",
       "      <th>test</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>log_loss</th>\n",
       "      <td>0.398210</td>\n",
       "      <td>0.387116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>accuracy</th>\n",
       "      <td>0.810000</td>\n",
       "      <td>0.843333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>balanced_accuracy</th>\n",
       "      <td>0.800987</td>\n",
       "      <td>0.871035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>precision</th>\n",
       "      <td>0.338983</td>\n",
       "      <td>0.397260</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall</th>\n",
       "      <td>0.789474</td>\n",
       "      <td>0.906250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>roc_auc</th>\n",
       "      <td>0.886956</td>\n",
       "      <td>0.938666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average_precision</th>\n",
       "      <td>0.678297</td>\n",
       "      <td>0.789859</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    roc_auc          \n",
       "                      train      test\n",
       "log_loss           0.398210  0.387116\n",
       "accuracy           0.810000  0.843333\n",
       "balanced_accuracy  0.800987  0.871035\n",
       "precision          0.338983  0.397260\n",
       "recall             0.789474  0.906250\n",
       "roc_auc            0.886956  0.938666\n",
       "average_precision  0.678297  0.789859"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "est = BrushClassifier(\n",
    "    functions=['SplitBest', 'Add', 'Sub', 'Mul', 'Div', 'Logabs', 'Exp'],\n",
    "    scorer='roc_auc',\n",
    "    pop_size=200,\n",
    "    max_gens=30,\n",
    "    max_depth=5,\n",
    "    max_size=20,\n",
    "    random_state=42,\n",
    "    verbosity=0,\n",
    ")\n",
    "est.fit(X_train, y_train)\n",
    "\n",
    "print('scorer:', est.parameters_.scorer)\n",
    "print('internal scorer value (train):', round(est.best_estimator_.fitness.loss, 4))\n",
    "print('model:', est.best_estimator_.get_model())\n",
    "\n",
    "before = report(est, 'roc_auc')\n",
    "before"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3aa6132b",
   "metadata": {},
   "source": [
    "## 3. Lock the internal nodes, switch to average precision, and refit\n",
    "\n",
    "A `lock_nodes_depth` larger than the tree depth locks every operator and split\n",
    "in the program. With `keep_leaves_unlocked=True`, the leaves (features and\n",
    "constants) stay free, so the search can still swap a leaf or grow it into a\n",
    "small subtree. With `keep_current_weights=False`, the weights are re-optimized.\n",
    "\n",
    "We then set `est.scorer = 'average_precision_score'` before calling\n",
    "`partial_fit`. The new scorer is picked up by the engine, so every program is\n",
    "re-evaluated, selected, and archived by average precision from here on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b33832db",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T19:12:14.025731Z",
     "iopub.status.busy": "2026-09-22T19:12:14.025682Z",
     "iopub.status.idle": "2026-09-22T19:12:20.981227Z",
     "shell.execute_reply": "2026-09-22T19:12:20.980941Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "scorer: average_precision_score\n",
      "internal scorer value (train): 0.917\n",
      "model before: Logistic(Add(0.98,Add(-1.05*Mul(-1.32*x_4,-1.01*x_2),x_0)))\n",
      "model after:  Logistic(Add(-2.77,Add(-1.53*Mul(x_4,x_2),3.87*Exp(0.42*x_0))))\n"
     ]
    }
   ],
   "source": [
    "structure_before = est.best_estimator_.get_model()\n",
    "dot_before = est.best_estimator_.get_model('dot')\n",
    "\n",
    "est.scorer = 'average_precision_score'\n",
    "est.partial_fit(\n",
    "    X_train, y_train,\n",
    "    lock_nodes_depth=est.max_depth + 1,  # deeper than any tree: lock every internal node\n",
    "    keep_leaves_unlocked=True,           # ...but leave the leaves free to change\n",
    "    keep_current_weights=False,          # weights can still be optimized\n",
    ")\n",
    "\n",
    "print('scorer:', est.parameters_.scorer)\n",
    "print('internal scorer value (train):', round(est.best_estimator_.fitness.loss, 4))\n",
    "print('model before:', structure_before)\n",
    "print('model after: ', est.best_estimator_.get_model())\n",
    "\n",
    "after = report(est, 'average_precision_score')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed1c72d3",
   "metadata": {},
   "source": [
    "The two programs side by side. The locked internal nodes are the same in\n",
    "both, and only the leaves (and the weights) differ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f7a39b67",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T19:12:20.982237Z",
     "iopub.status.busy": "2026-09-22T19:12:20.982178Z",
     "iopub.status.idle": "2026-09-22T19:12:21.166282Z",
     "shell.execute_reply": "2026-09-22T19:12:21.165925Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "<div style=\"display: flex; gap: 2em; align-items: flex-start; flex-wrap: wrap;\">\n",
       "  <div><h4>Before (AUROC)</h4><?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 14.1.2 (0)\n",
       " -->\n",
       "<!-- Title: G Pages: 1 -->\n",
       "<svg width=\"206pt\" height=\"386pt\"\n",
       " viewBox=\"0.00 0.00 206.00 386.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 382.25)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-382.25 202.3,-382.25 202.3,4 -4,4\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"99.15\" y=\"-5.75\" font-family=\"Times,serif\" font-size=\"10.00\">^ split feature fixed, * split threshold fixed</text>\n",
       "<!-- 1473c7000 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>1473c7000</title>\n",
       "<ellipse fill=\"lightcoral\" stroke=\"black\" cx=\"135\" cy=\"-360.25\" rx=\"41.12\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"135\" y=\"-355.2\" font-family=\"Times,serif\" font-size=\"14.00\">Logistic</text>\n",
       "</g>\n",
       "<!-- 105024bc0 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>105024bc0</title>\n",
       "<ellipse fill=\"lightcoral\" stroke=\"black\" cx=\"135\" cy=\"-287.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"135\" y=\"-282.2\" font-family=\"Times,serif\" font-size=\"14.00\">Add</text>\n",
       "</g>\n",
       "<!-- 1473c7000&#45;&gt;105024bc0 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>1473c7000&#45;&gt;105024bc0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M135,-342.06C135,-334.48 135,-325.35 135,-316.79\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"138.5,-316.79 135,-306.79 131.5,-316.79 138.5,-316.79\"/>\n",
       "</g>\n",
       "<!-- 1473c70b0 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>1473c70b0</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-214.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"99\" y=\"-209.2\" font-family=\"Times,serif\" font-size=\"14.00\">Add</text>\n",
       "</g>\n",
       "<!-- 105024bc0&#45;&gt;1473c70b0 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>105024bc0&#45;&gt;1473c70b0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M126.65,-269.78C122.42,-261.45 117.19,-251.13 112.42,-241.72\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"115.63,-240.31 107.99,-232.98 109.39,-243.48 115.63,-240.31\"/>\n",
       "</g>\n",
       "<!-- 105024bc0Offset -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>105024bc0Offset</title>\n",
       "<ellipse fill=\"lightcoral\" stroke=\"black\" cx=\"171\" cy=\"-214.25\" rx=\"27.3\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"171\" y=\"-209.2\" font-family=\"Times,serif\" font-size=\"14.00\">0.98</text>\n",
       "</g>\n",
       "<!-- 105024bc0&#45;&gt;105024bc0Offset -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>105024bc0&#45;&gt;105024bc0Offset</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M143.35,-269.78C147.58,-261.45 152.81,-251.13 157.58,-241.72\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"160.61,-243.48 162.01,-232.98 154.37,-240.31 160.61,-243.48\"/>\n",
       "</g>\n",
       "<!-- 1473884b0 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>1473884b0</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"63\" cy=\"-125.75\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"63\" y=\"-120.7\" font-family=\"Times,serif\" font-size=\"14.00\">Mul</text>\n",
       "</g>\n",
       "<!-- 1473c70b0&#45;&gt;1473884b0 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>1473c70b0&#45;&gt;1473884b0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M92.06,-196.57C87.03,-184.49 80.12,-167.88 74.3,-153.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"77.58,-152.66 70.5,-144.77 71.11,-155.35 77.58,-152.66\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"98.47\" y=\"-164.95\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1.05</text>\n",
       "</g>\n",
       "<!-- x_0 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>x_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"135\" cy=\"-125.75\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"135\" y=\"-120.7\" font-family=\"Times,serif\" font-size=\"14.00\">x_0</text>\n",
       "</g>\n",
       "<!-- 1473c70b0&#45;&gt;x_0 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1473c70b0&#45;&gt;x_0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M107.9,-197.09C110.97,-191.24 114.3,-184.53 117,-178.25 120.26,-170.65 123.4,-162.24 126.11,-154.46\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"129.38,-155.7 129.26,-145.11 122.75,-153.47 129.38,-155.7\"/>\n",
       "</g>\n",
       "<!-- x_4 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>x_4</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-37.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"27\" y=\"-32.2\" font-family=\"Times,serif\" font-size=\"14.00\">x_4</text>\n",
       "</g>\n",
       "<!-- 1473884b0&#45;&gt;x_4 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>1473884b0&#45;&gt;x_4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M56.06,-108.07C51.03,-95.99 44.12,-79.38 38.3,-65.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"41.58,-64.16 34.5,-56.27 35.11,-66.85 41.58,-64.16\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"62.47\" y=\"-76.45\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1.32</text>\n",
       "</g>\n",
       "<!-- x_2 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>x_2</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-37.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"99\" y=\"-32.2\" font-family=\"Times,serif\" font-size=\"14.00\">x_2</text>\n",
       "</g>\n",
       "<!-- 1473884b0&#45;&gt;x_2 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>1473884b0&#45;&gt;x_2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M69.94,-108.07C74.97,-95.99 81.88,-79.38 87.7,-65.39\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"90.89,-66.85 91.5,-56.27 84.42,-64.16 90.89,-66.85\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"98.47\" y=\"-76.45\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1.01</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n",
       "</div>\n",
       "  <div><h4>After (average precision)</h4><?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 14.1.2 (0)\n",
       " -->\n",
       "<!-- Title: G Pages: 1 -->\n",
       "<svg width=\"248pt\" height=\"386pt\"\n",
       " viewBox=\"0.00 0.00 248.00 386.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 382.25)\">\n",
       "<title>G</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-382.25 244.37,-382.25 244.37,4 -4,4\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"120.18\" y=\"-5.75\" font-family=\"Times,serif\" font-size=\"10.00\">^ split feature fixed, * split threshold fixed</text>\n",
       "<!-- 15f54b570 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>15f54b570</title>\n",
       "<ellipse fill=\"lightcoral\" stroke=\"black\" cx=\"172\" cy=\"-360.25\" rx=\"41.12\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"172\" y=\"-355.2\" font-family=\"Times,serif\" font-size=\"14.00\">Logistic</text>\n",
       "</g>\n",
       "<!-- 15f54ba00 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>15f54ba00</title>\n",
       "<ellipse fill=\"lightcoral\" stroke=\"black\" cx=\"172\" cy=\"-287.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"172\" y=\"-282.2\" font-family=\"Times,serif\" font-size=\"14.00\">Add</text>\n",
       "</g>\n",
       "<!-- 15f54b570&#45;&gt;15f54ba00 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>15f54b570&#45;&gt;15f54ba00</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M172,-342.06C172,-334.48 172,-325.35 172,-316.79\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"175.5,-316.79 172,-306.79 168.5,-316.79 175.5,-316.79\"/>\n",
       "</g>\n",
       "<!-- 15f5c09c0 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>15f5c09c0</title>\n",
       "<ellipse fill=\"lightcoral\" stroke=\"black\" cx=\"135\" cy=\"-214.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"135\" y=\"-209.2\" font-family=\"Times,serif\" font-size=\"14.00\">Add</text>\n",
       "</g>\n",
       "<!-- 15f54ba00&#45;&gt;15f5c09c0 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>15f54ba00&#45;&gt;15f5c09c0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M163.42,-269.78C159.08,-261.45 153.7,-251.13 148.79,-241.72\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"151.96,-240.22 144.23,-232.97 145.75,-243.46 151.96,-240.22\"/>\n",
       "</g>\n",
       "<!-- 15f54ba00Offset -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>15f54ba00Offset</title>\n",
       "<ellipse fill=\"lightcoral\" stroke=\"black\" cx=\"210\" cy=\"-214.25\" rx=\"30.37\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"210\" y=\"-209.2\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;2.77</text>\n",
       "</g>\n",
       "<!-- 15f54ba00&#45;&gt;15f54ba00Offset -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>15f54ba00&#45;&gt;15f54ba00Offset</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M180.81,-269.78C185.27,-261.45 190.8,-251.13 195.84,-241.72\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"198.89,-243.43 200.52,-232.96 192.72,-240.13 198.89,-243.43\"/>\n",
       "</g>\n",
       "<!-- 15f5c0a70 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>15f5c0a70</title>\n",
       "<ellipse fill=\"lightcoral\" stroke=\"black\" cx=\"99\" cy=\"-125.75\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"99\" y=\"-120.7\" font-family=\"Times,serif\" font-size=\"14.00\">Mul</text>\n",
       "</g>\n",
       "<!-- 15f5c09c0&#45;&gt;15f5c0a70 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>15f5c09c0&#45;&gt;15f5c0a70</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M125.18,-197.21C121.85,-191.37 118.27,-184.63 115.5,-178.25 112.24,-170.73 109.28,-162.34 106.8,-154.56\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"110.22,-153.76 103.97,-145.21 103.52,-155.79 110.22,-153.76\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"129.75\" y=\"-164.95\" font-family=\"Times,serif\" font-size=\"14.00\">&#45;1.53</text>\n",
       "</g>\n",
       "<!-- 15f5f0460 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>15f5f0460</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"171\" cy=\"-125.75\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"171\" y=\"-120.7\" font-family=\"Times,serif\" font-size=\"14.00\">Exp</text>\n",
       "</g>\n",
       "<!-- 15f5c09c0&#45;&gt;15f5f0460 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>15f5c09c0&#45;&gt;15f5f0460</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M141.94,-196.57C146.97,-184.49 153.88,-167.88 159.7,-153.89\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"162.89,-155.35 163.5,-144.77 156.42,-152.66 162.89,-155.35\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"168.22\" y=\"-164.95\" font-family=\"Times,serif\" font-size=\"14.00\">3.87</text>\n",
       "</g>\n",
       "<!-- x_4 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>x_4</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"27\" cy=\"-37.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"27\" y=\"-32.2\" font-family=\"Times,serif\" font-size=\"14.00\">x_4</text>\n",
       "</g>\n",
       "<!-- 15f5c0a70&#45;&gt;x_4 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>15f5c0a70&#45;&gt;x_4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M86.46,-109.68C75.48,-96.49 59.34,-77.11 46.65,-61.86\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"49.7,-60.04 40.61,-54.6 44.32,-64.52 49.7,-60.04\"/>\n",
       "</g>\n",
       "<!-- x_2 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>x_2</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"99\" cy=\"-37.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"99\" y=\"-32.2\" font-family=\"Times,serif\" font-size=\"14.00\">x_2</text>\n",
       "</g>\n",
       "<!-- 15f5c0a70&#45;&gt;x_2 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>15f5c0a70&#45;&gt;x_2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M99,-107.66C99,-96.01 99,-80.3 99,-66.77\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"102.5,-67.11 99,-57.11 95.5,-67.11 102.5,-67.11\"/>\n",
       "</g>\n",
       "<!-- x_0 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>x_0</title>\n",
       "<ellipse fill=\"none\" stroke=\"black\" cx=\"171\" cy=\"-37.25\" rx=\"27\" ry=\"18\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"171\" y=\"-32.2\" font-family=\"Times,serif\" font-size=\"14.00\">x_0</text>\n",
       "</g>\n",
       "<!-- 15f5f0460&#45;&gt;x_0 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>15f5f0460&#45;&gt;x_0</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M171,-107.66C171,-96.01 171,-80.3 171,-66.77\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"174.5,-67.11 171,-57.11 167.5,-67.11 174.5,-67.11\"/>\n",
       "<text xml:space=\"preserve\" text-anchor=\"middle\" x=\"183\" y=\"-76.45\" font-family=\"Times,serif\" font-size=\"14.00\">0.42</text>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n",
       "</div>\n",
       "</div>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "def svg(dot):\n",
    "    return graphviz.Source(dot).pipe(format='svg').decode()\n",
    "\n",
    "HTML(f\"\"\"\n",
    "<div style=\"display: flex; gap: 2em; align-items: flex-start; flex-wrap: wrap;\">\n",
    "  <div><h4>Before (AUROC)</h4>{svg(dot_before)}</div>\n",
    "  <div><h4>After (average precision)</h4>{svg(est.best_estimator_.get_model('dot'))}</div>\n",
    "</div>\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26aa005f",
   "metadata": {},
   "source": [
    "## 4. Compare every metric before and after the switch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "32d527ab",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-09-22T19:12:21.167352Z",
     "iopub.status.busy": "2026-09-22T19:12:21.167284Z",
     "iopub.status.idle": "2026-09-22T19:12:21.173975Z",
     "shell.execute_reply": "2026-09-22T19:12:21.173673Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">roc_auc</th>\n",
       "      <th colspan=\"2\" halign=\"left\">average_precision_score</th>\n",
       "      <th colspan=\"2\" halign=\"left\">delta</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>train</th>\n",
       "      <th>test</th>\n",
       "      <th>train</th>\n",
       "      <th>test</th>\n",
       "      <th>train</th>\n",
       "      <th>test</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>log_loss</th>\n",
       "      <td>0.3982</td>\n",
       "      <td>0.3871</td>\n",
       "      <td>0.3819</td>\n",
       "      <td>0.3787</td>\n",
       "      <td>-0.0163</td>\n",
       "      <td>-0.0084</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>accuracy</th>\n",
       "      <td>0.8100</td>\n",
       "      <td>0.8433</td>\n",
       "      <td>0.8400</td>\n",
       "      <td>0.8633</td>\n",
       "      <td>0.0300</td>\n",
       "      <td>0.0200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>balanced_accuracy</th>\n",
       "      <td>0.8010</td>\n",
       "      <td>0.8710</td>\n",
       "      <td>0.8178</td>\n",
       "      <td>0.8822</td>\n",
       "      <td>0.0168</td>\n",
       "      <td>0.0112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>precision</th>\n",
       "      <td>0.3390</td>\n",
       "      <td>0.3973</td>\n",
       "      <td>0.3846</td>\n",
       "      <td>0.4328</td>\n",
       "      <td>0.0456</td>\n",
       "      <td>0.0356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>recall</th>\n",
       "      <td>0.7895</td>\n",
       "      <td>0.9062</td>\n",
       "      <td>0.7895</td>\n",
       "      <td>0.9062</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.0000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>roc_auc</th>\n",
       "      <td>0.8870</td>\n",
       "      <td>0.9387</td>\n",
       "      <td>0.8892</td>\n",
       "      <td>0.9397</td>\n",
       "      <td>0.0022</td>\n",
       "      <td>0.0010</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>average_precision</th>\n",
       "      <td>0.6783</td>\n",
       "      <td>0.7899</td>\n",
       "      <td>0.7041</td>\n",
       "      <td>0.7983</td>\n",
       "      <td>0.0258</td>\n",
       "      <td>0.0085</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  roc_auc         average_precision_score           delta  \\\n",
       "                    train    test                   train    test   train   \n",
       "log_loss           0.3982  0.3871                  0.3819  0.3787 -0.0163   \n",
       "accuracy           0.8100  0.8433                  0.8400  0.8633  0.0300   \n",
       "balanced_accuracy  0.8010  0.8710                  0.8178  0.8822  0.0168   \n",
       "precision          0.3390  0.3973                  0.3846  0.4328  0.0456   \n",
       "recall             0.7895  0.9062                  0.7895  0.9062  0.0000   \n",
       "roc_auc            0.8870  0.9387                  0.8892  0.9397  0.0022   \n",
       "average_precision  0.6783  0.7899                  0.7041  0.7983  0.0258   \n",
       "\n",
       "                           \n",
       "                     test  \n",
       "log_loss          -0.0084  \n",
       "accuracy           0.0200  \n",
       "balanced_accuracy  0.0112  \n",
       "precision          0.0356  \n",
       "recall             0.0000  \n",
       "roc_auc            0.0010  \n",
       "average_precision  0.0085  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comparison = pd.concat([before, after], axis=1)\n",
    "comparison[('delta', 'train')] = comparison[('average_precision_score', 'train')] - comparison[('roc_auc', 'train')]\n",
    "comparison[('delta', 'test')]  = comparison[('average_precision_score', 'test')]  - comparison[('roc_auc', 'test')]\n",
    "comparison.round(4)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "brush",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}