Logging the evolution

Set the logfile parameter to a path, and every call to fit will write:

file

content

<logfile>

one row per generation: losses, sizes, stall count, archive size, number of evaluations, and the best expression

<logfile>_islands.csv

the same statistics, one row per island and generation

<logfile>_simplifications.csv

every replacement performed by the simplifiers (original → replacement)

<logfile>_simplification_table

the expressions stored by the inexact simplifier, at the end of the run (only with inexact_simplification=True)

<logfile>_runs.jsonl

one JSON object per run, with the seed and all parameters

All files are valid CSV (expressions are quoted) and have a run_id column. Files are opened in append mode, so fitting several times with the same logfile adds rows instead of overwriting them. Brush refuses to append to a file that was written with different columns (for example, by an older version).

import os, shutil

import pandas as pd
import matplotlib.pyplot as plt

from pybrush import BrushRegressor

LOG_DIR = "logs"
shutil.rmtree(LOG_DIR, ignore_errors=True)
os.makedirs(LOG_DIR, exist_ok=True)

df = pd.read_csv("../examples/datasets/d_enc.csv")
X = df.drop(columns="label")
y = df["label"]

pd.set_option("display.max_colwidth", 100)

Generating the log files

We fit three seeds and write all of them to the same logfile; the run_id column tells the runs apart.

LOGFILE = os.path.join(LOG_DIR, "brush.csv")
seeds = [0, 1, 2]
estimators = {}

for seed in seeds:
    est = BrushRegressor(
        functions=["Add", "Sub", "Mul", "Div", "Sin", "Cos", "Exp", "Logabs", "Sqrtabs"],
        max_gens=50,
        pop_size=100,
        num_islands=4,
        max_size=50,
        max_depth=6,
        objectives=["scorer", "linear_complexity"],
        inexact_simplification=True,
        logfile=LOGFILE, # This toggle on the generation of logfiles
        random_state=seed, # Brush has total random seed control
        verbosity=0,
    )
    est.fit(X, y)
    estimators[seed] = est

# Files were generated after fitting the estimator
sorted(os.listdir(LOG_DIR))
['brush.csv',
 'brush.csv_islands.csv',
 'brush.csv_runs.jsonl',
 'brush.csv_simplification_table',
 'brush.csv_simplifications.csv']

Run metadata

_runs.jsonl has one line per call to fit, with the full set of parameters.

runs = pd.json_normalize(pd.read_json(LOGFILE + "_runs.jsonl", lines=True).to_dict("records"))
runs[["run_id", "random_state", "params.pop_size", "params.max_gens", "params.num_islands", "params.scorer"]]
run_id random_state params.pop_size params.max_gens params.num_islands params.scorer
0 20260923T121716.761-c517 0 100 50 4 mse
1 20260923T121720.173-a13f 1 100 50 4 mse
2 20260923T121723.647-be9d 2 100 50 4 mse

The evolution log

Each row is one generation of one run. best_* refer to the best individual (chosen on the validation partition), and med_*/max_* summarize the whole population.

n_evaluations is cumulative. Models that survived across generations are not refitted thus not counted towards the number of evaluations. The number of evaluations refers to the number of times a program was fitted (had it’s weights optimized). It does not reflect the number of optimization steps performed for each individual.

log = pd.read_csv(LOGFILE)

print("columns:", list(log.columns))
log.head()
columns: ['run_id', 'random_state', 'generation', 'time', 'best_score', 'best_score_val', 'med_score', 'med_score_val', 'med_size', 'med_complexity', 'max_size', 'max_complexity', 'best_size', 'best_complexity', 'stall_count', 'archive_size', 'n_evaluations', 'best_model']
run_id random_state generation time best_score best_score_val med_score med_score_val med_size med_complexity max_size max_complexity best_size best_complexity stall_count archive_size n_evaluations best_model
0 20260923T121716.761-c517 0 0 0.068071 10.350908 9.278577 71.704758 77.782562 7 32 39 37237 39 5225 0 5 400 1.62*Add(0.32*Add(109.51*Div(109.30*x3,200.26*Add(225.90*x2,-788.44*Sqrtabs(-3332.07*x4))),31.95...
1 20260923T121716.761-c517 0 1 0.126026 9.956403 9.127022 20.323715 21.707689 6 25 42 37237 39 5225 0 7 700 1.62*Add(0.32*Add(67.72*Div(67.59*x3,302.11*Add(227.53*x2,-783.07*Sqrtabs(-3286.79*x4))),32.25*x...
2 20260923T121716.761-c517 0 2 0.189985 9.936137 9.127022 17.851387 19.497444 7 27 45 8990 39 5225 1 7 1000 1.62*Add(0.32*Add(67.72*Div(67.59*x3,302.11*Add(227.53*x2,-783.07*Sqrtabs(-3286.79*x4))),32.25*x...
3 20260923T121716.761-c517 0 3 0.244779 9.934284 9.127022 15.598775 16.937330 13 95 45 8990 39 5225 2 9 1300 1.62*Add(0.32*Add(67.72*Div(67.59*x3,302.11*Add(227.53*x2,-783.07*Sqrtabs(-3286.79*x4))),32.25*x...
4 20260923T121716.761-c517 0 4 0.302656 9.932689 9.127022 15.123131 16.591620 13 107 43 8990 39 5225 3 10 1600 1.62*Add(0.32*Add(67.72*Div(67.59*x3,302.11*Add(227.53*x2,-783.07*Sqrtabs(-3286.79*x4))),32.25*x...
# number of generations, best metric and final expression of each run
summary = log.groupby("random_state").agg(
    n_generations=("generation", "count"),
    total_time=("time", "last"),
    n_evaluations=("n_evaluations", "last"),
    best_score=("best_score", "last"),
    best_score_val=("best_score_val", "last"),
    archive_size=("archive_size", "last"),
    best_model=("best_model", "last"),
)
summary
n_generations total_time n_evaluations best_score best_score_val archive_size best_model
random_state
0 50 3.390770 15100 5.959468 6.035597 15 Add(0.19*Add(812.02*Div(6852.51*x0,4929.73*Add(185.08*x2,-422.56*Sqrtabs(-3222.37*x4))),82.75*x6...
1 50 3.447640 15100 3.812312 4.720665 13 Div(1.39*Exp(0.17*Sin(0.97*Sub(1.84*Add(1.00*x2,-13.06),0.96*x1))),0.56*Div(Sub(0.97*Sqrtabs(Sqr...
2 50 3.329469 15100 4.000235 4.290797 15 0.00*Sub(Add(24.98*Mul(2.71*x4,14.29*x6),1387.94*Cos(1.00*Sub(1.56*x4,1.00*Mul(x2,1.00*x1)))),0....

Convergence

fig, axs = plt.subplots(1, 2, figsize=(11, 4), sharey=True)

for seed, l in log.groupby("random_state"):
    axs[0].plot(l["generation"], l["best_score"], label=f"seed {seed}")
    axs[1].plot(l["generation"], l["best_score_val"], label=f"seed {seed}")

axs[0].set_title("Best individual - train loss")
axs[1].set_title("Best individual - validation loss")
for ax in axs:
    ax.set_xlabel("generation")
    ax.set_yscale("log")
    ax.grid(alpha=0.3)
axs[0].set_ylabel("loss (MSE)")
axs[1].legend()
plt.tight_layout()
../_images/3db70a3c63a7ea302040f0d204441d4da8e4cb75be216b066d537f0a3e2deef7.png
l = log[log["random_state"] == 0]
fig, axs = plt.subplots(2, 2, figsize=(11, 7))

axs[0, 0].plot(l["generation"], l["best_score"], label="best")
axs[0, 0].plot(l["generation"], l["med_score"], label="median")
axs[0, 0].set_yscale("log")
axs[0, 0].set_title("Train loss")
axs[0, 0].legend()

axs[0, 1].plot(l["generation"], l["med_size"], label="median")
axs[0, 1].plot(l["generation"], l["max_size"], label="max")
axs[0, 1].plot(l["generation"], l["best_size"], label="best individual")
axs[0, 1].set_title("Program size")
axs[0, 1].legend()

axs[1, 0].plot(l["generation"], l["stall_count"], drawstyle="steps-post")
axs[1, 0].set_title("Generations without improving the best individual")

axs[1, 1].plot(l["generation"], l["archive_size"], drawstyle="steps-post")
axs[1, 1].set_title("Archive (Pareto front) size")

for ax in axs.flat:
    ax.set_xlabel("generation")
    ax.grid(alpha=0.3)
fig.suptitle("seed 0")
plt.tight_layout()
../_images/56918a1347bbf4f2ac32efc0a5ab886ba1b770b439f88ad3a945d5de7b74070a.png
# cost of the search
fig, axs = plt.subplots(1, 2, figsize=(11, 3.5))
for seed, l in log.groupby("random_state"):
    axs[0].plot(l["generation"], l["n_evaluations"], label=f"seed {seed}")
    axs[1].plot(l["n_evaluations"], l["best_score_val"], label=f"seed {seed}")
axs[0].set_xlabel("generation")
axs[0].set_ylabel("cumulative evaluations")
axs[1].set_xlabel("cumulative evaluations")
axs[1].set_ylabel("best validation loss")
axs[1].set_yscale("log")
for ax in axs:
    ax.grid(alpha=0.3)
axs[1].legend()
plt.tight_layout()
../_images/babd0f3d44c513b956f5b119fb9f22a16fbc6b08d7adcef687c00e9396336c3d.png

Best expression over the generations

The best_model column shows how the best individual changed. We keep only the generations where it changed.

l = log[log["random_state"] == 0]
changes = l[l["best_model"] != l["best_model"].shift()]
changes[["generation", "best_score_val", "best_size", "best_model"]]
generation best_score_val best_size best_model
0 0 9.278577 39 1.62*Add(0.32*Add(109.51*Div(109.30*x3,200.26*Add(225.90*x2,-788.44*Sqrtabs(-3332.07*x4))),31.95...
1 1 9.127022 39 1.62*Add(0.32*Add(67.72*Div(67.59*x3,302.11*Add(227.53*x2,-783.07*Sqrtabs(-3286.79*x4))),32.25*x...
8 8 8.810316 45 0.69*Add(0.43*Add(970.14*Div(48.11*x3,-381.89*Add(230.46*x2,-762.89*Sqrtabs(-3304.37*x4))),55.19...
9 9 8.810287 43 0.70*Add(23.25*Add(332.12*Div(16.47*x3,-2464.26*Add(231.03*x2,-763.86*Sqrtabs(-3312.69*x4))),x6)...
10 10 8.810237 45 0.68*Add(0.43*Add(965.64*Div(48.34*x3,-383.90*Add(230.41*x2,-762.89*Sqrtabs(-3302.95*x4))),54.92...
11 11 7.319740 45 0.68*Add(0.43*Add(2659.25*Div(16753.25*x0,3640.52*Add(185.45*x2,-421.66*Sqrtabs(-3207.36*x4))),5...
12 12 6.632836 45 0.67*Add(0.41*Add(1756.35*Div(11066.24*x0,5321.64*Add(185.13*x2,-422.40*Sqrtabs(-3218.61*x4))),5...
13 13 6.288856 45 0.68*Add(0.40*Add(1699.36*Div(10700.80*x0,5499.05*Add(185.11*x2,-422.43*Sqrtabs(-3219.07*x4))),5...
14 14 6.271890 45 0.36*Add(4.50*Add(-1149.91*Div(-5750.37*x0,11909.45*Add(185.11*x2,-422.35*Sqrtabs(-3220.11*x4)))...
15 15 6.192916 49 0.45*Add(0.22*Add(2058.68*Div(16604.21*x0,4962.30*Add(185.08*x2,-422.45*Sqrtabs(-3219.49*x4))),0...
25 25 6.168351 49 0.27*Add(0.54*Add(1827.21*Div(12099.95*x0,4751.91*Add(185.10*x2,-422.49*Sqrtabs(-3219.78*x4))),1...
29 29 6.158276 49 0.22*Add(0.57*Add(1877.77*Div(12344.38*x0,4546.73*Add(185.09*x2,-422.49*Sqrtabs(-3219.77*x4))),1...
31 31 6.138827 48 1.92*Add(0.17*Add(1274.97*Div(9031.33*x0,5986.33*Add(185.10*x2,-422.48*Sqrtabs(-3220.48*x4))),4....
46 46 6.138803 42 1.54*Add(0.20*Add(1276.03*Div(9249.64*x0,5857.35*Add(185.10*x2,-422.48*Sqrtabs(-3220.58*x4))),53...
48 48 6.035705 50 1.75*Add(0.16*Add(892.99*Div(6962.25*x0,8090.15*Add(185.08*x2,-422.56*Sqrtabs(-3222.30*x4))),56....
49 49 6.035597 48 Add(0.19*Add(812.02*Div(6852.51*x0,4929.73*Add(185.08*x2,-422.56*Sqrtabs(-3222.37*x4))),82.75*x6...

Per-island statistics

islands = pd.read_csv(LOGFILE + "_islands.csv")
print("columns:", list(islands.columns))

isl = islands[islands["run_id"] == runs.loc[runs["random_state"] == 0, "run_id"].iloc[0]]

fig, axs = plt.subplots(1, 2, figsize=(11, 4))
for island, g in isl.groupby("island"):
    axs[0].plot(g["generation"], g["best_score"], label=f"island {island}")
    axs[1].plot(g["generation"], g["med_size"], label=f"island {island}")
axs[0].set_title("Best train loss per island (seed 0)")
axs[0].set_yscale("log")
axs[1].set_title("Median size per island (seed 0)")
for ax in axs:
    ax.set_xlabel("generation")
    ax.grid(alpha=0.3)
axs[1].legend()
plt.tight_layout()
columns: ['run_id', 'generation', 'island', 'n_individuals', 'best_score', 'best_score_val', 'med_score', 'med_score_val', 'med_size', 'med_complexity', 'max_size', 'max_complexity']
../_images/37356d7bd4dd9557880fcc201868f641fd5c1c6e89dfb9de122d4eda29089cab.png

Final Pareto fronts

The CSV logs only keep the best individual. The whole front (extracted right before the last valid generation) is available from the fitted estimator (archive_).

front = pd.DataFrame([
    {
        "size": ind.program.size(),
        "complexity": ind.program.linear_complexity(),
        "val_loss": ind.fitness.loss_v,
        "model": ind.program.get_model(),
    }
    for ind in estimators[0].archive_
]).sort_values("complexity")
front
size complexity val_loss model
0 1 2 97.703964 24.17
1 3 5 19.342838 4.60*x4
2 7 13 14.473068 11.41*Add(0.37*x4,x6)
3 10 20 12.181124 6.69*Exp(Sub(0.21*x4,-0.63*x6))
4 11 21 10.951211 Sub(5.61*x4,Sub(-17.91*x6,-11.82*x0))
5 11 25 10.312360 Sub(-10.30,0.01*Mul(Sub(-338.44*x6,x1),x4))
6 13 28 9.265789 Sub(-0.03*x3,0.01*Mul(Sub(-380.14*x6,x1),x4))
7 15 30 9.265786 Sub(-0.03*x3,0.07*Mul(Sub(-42.68*x6,0.11*x1),x4))
8 17 32 9.265783 Sub(-0.03*x3,0.09*Mul(Sub(-55.68*x6,0.15*x1),0.61*x4))
9 23 45 8.871875 0.15*Sub(-1217.18,Mul(2.08*Add(Sub(-56.72*x6,1.17*x1),0.21*x3),Sub(x0,-0.02*x4)))
10 27 49 8.869665 0.22*Sub(-837.32,Mul(0.84*Add(Sub(-24.64*x6,0.51*x1),0.09*x3),0.44*Sub(8.90*x0,-0.21*x4)))
11 32 60 7.998517 0.66*Add(0.02*Add(31.71*Div(1.43*x2,Add(185.08*x2,-422.53*Sqrtabs(-3221.28*x4))),1048.94*x6),Add...
12 36 68 6.334166 3.00*Add(0.37*Add(965.81*Div(8205.08*x0,8424.49*Add(185.03*x2,-422.31*Sqrtabs(-3216.79*x4))),14....
13 40 75 6.138934 1.46*Add(Add(729.70*Div(5337.57*x0,9268.67*Add(185.10*x2,-422.49*Sqrtabs(-3220.58*x4))),11.07*x6...
14 48 87 6.035597 Add(0.19*Add(812.02*Div(6852.51*x0,4929.73*Add(185.08*x2,-422.56*Sqrtabs(-3222.37*x4))),82.75*x6...
fig, ax = plt.subplots(figsize=(6, 4))
for seed, e in estimators.items():
    pts = sorted((i.program.linear_complexity(), i.fitness.loss_v) for i in e.archive_)
    ax.step(*zip(*pts), where="post", marker="o", label=f"seed {seed}")
ax.set_xscale("log")
ax.set_yscale("log")
ax.xaxis.set_minor_formatter(plt.NullFormatter())
ax.set_xlabel("linear complexity")
ax.set_ylabel("validation loss")
ax.set_title("Final Pareto fronts")
ax.legend()
ax.grid(alpha=0.3)
../_images/3a9ab23cd2ba8530ead1f76803735a4e0fb8a90a48fd6566f87d3d2107c078a3.png

Simplifications

Each row of _simplifications.csv is one replacement made in an offspring:

  • constants: a subtree with (almost) constant output was replaced by a constant;

  • inexact: a subtree was replaced by a smaller one with (almost) the same output, found with locality-sensitive hashing. distance is the MSE between the predictions of the program before and after the replacement.

simp = pd.read_csv(LOGFILE + "_simplifications.csv")
print("columns:", list(simp.columns))
simp.groupby(["run_id", "simplifier"]).size().unstack(fill_value=0)
columns: ['run_id', 'generation', 'individual_id', 'simplifier', 'ret_type', 'original', 'replacement', 'distance']
simplifier constants inexact
run_id
20260923T121716.761-c517 244 102
20260923T121720.173-a13f 180 98
20260923T121723.647-be9d 175 114
counts = simp.groupby(["generation", "simplifier"]).size().unstack(fill_value=0)
counts = counts.reindex(range(log["generation"].max() + 1), fill_value=0)
ax = counts.plot(kind="bar", stacked=True, figsize=(11, 3.5), width=0.9)
ax.set_ylabel("replacements (all runs)")
ax.set_xticks(range(0, len(counts), 5), labels=range(0, len(counts), 5), rotation=0)
ax.grid(alpha=0.3, axis="y")
plt.tight_layout()
../_images/b2ad3042f3682245dac18cb728ab91a3695a101f6e39e1565472d70089045139.png

Some examples

The inexact replacements that removed the most (by expression length):

inexact = simp[simp["simplifier"] == "inexact"].copy()
inexact["saved_chars"] = inexact["original"].str.len() - inexact["replacement"].str.len()
inexact.sort_values("saved_chars", ascending=False)[
    ["generation", "individual_id", "original", "replacement", "distance"]
].head(10)
generation individual_id original replacement distance
406 23 2404 Add(620.09*Exp(-9071.97*Div(-9068.50*x0,-19514.49*x3)),2.01*Cos(0.96*Add(0.96*x3,9.61*x6))) 2.01*Cos(0.96*Add(0.96*x3,9.61*x6)) 2.754639e-06
477 31 3281 Add(0.01*Mul(0.02*x4,0.05*Add(-433.39*x6,0.05*x2)),-1.12*Sin(-46.28*x0)) -1.12*Sin(-46.28*x0) 4.804618e-06
747 26 2754 1.00*Sub(1.00*x2,1.14*Sub(3.65*x0,-1.01*Logabs(-0.63*x4))) Sub(x2,x4) 4.830749e-06
593 46 4727 Add(Mul(-0.00*Sub(0.00*x1,4.43*x6),0.00*x3),x0) x0 1.637908e-06
618 49 5057 1.87*Div(Sub(Sqrtabs(2.10*x0),0.01*x6),Add(1.74*x0,-1.23*x0)) Div(Logabs(x1),1.00) 1.941102e-06
377 2 368 0.97*Add(0.70*Sqrtabs(0.37*x0),1.06*Add(1.12*x4,1.12*x4)) Add(1.22*x4,1.22*x4) 3.768802e-06
870 43 4428 1.26*Sub(1.93*x4,3.17*Div(x2,-2.06*x1)) x4 4.058915e-06
634 0 157 Add(-2373540.50*Cos(5477914.00*x4),-2411207.25*Sub(5827149.00*x4,110628.20*x1)) -2411207.25*Sub(5827149.00*x4,110628.20*x1) 1.877450e-16
61 11 1207 0.95*Add(0.97*Div(0.97*x4,1.03*x0),0.95*Div(0.95*x3,1.05*x0)) 2.13*Div(2.13*x3,0.31*x0) 4.574575e-08
843 39 4060 0.81*Add(0.47*Mul(0.47*x3,0.13),0.73*x2) Sub(x2,x4) 7.030044e-06
simp[simp["simplifier"] == "constants"][["generation", "original", "replacement"]].head(10)
generation original replacement
0 0 Exp(Mul(Add(Cos(x3),Add(x2,x6)),Sub(1.00,373.44*Sqrtabs(555.29*x4)))) 0.00
1 0 -2.98*Div(-2.92*x0,5.92*x0) 1.47
2 0 0.13*Mul(0.06,0.19*x0) 0.00
3 0 -0.23*Mul(-1.27,-0.23) -0.07
4 0 -38.37*Sin(-5.00) -36.82
5 0 Sub(x2,x2) 0.00
6 0 1.50*Sin(1.69) 1.49
7 0 2.51*Exp(Mul(Exp(x0),Mul(Mul(x1,x2),Sin(Logabs(x2))))) 0.00
8 0 12.85*Logabs(31326.93) 133.03
9 0 Sub(x2,x2) 0.00

The replaced subtrees are not identical: the inexact simplifier compares centered predictions, so a replacement may differ by an offset or scale that the program’s weights absorb. distance records how little the predictions of the whole program changed.

The simplification table

_simplification_table is a dump of every expression stored by the inexact simplifier at the end of each run. Expressions sharing (DataType, Plane, Key) are candidates to replace each other.

table = pd.read_csv(LOGFILE + "_simplification_table")
print("columns:", list(table.columns))
table.groupby(["run_id", "DataType"]).size().unstack(fill_value=0)
columns: ['run_id', 'DataType', 'Plane', 'Key', 'Tree']
DataType ArrayB ArrayF ArrayI
run_id
20260923T121716.761-c517 1 2758 3
20260923T121720.173-a13f 1 3062 3
20260923T121723.647-be9d 1 2619 3