Working with Programs
In Brush, a Program is an executable data structure.
You may think of it as a model or a function mapping feature inputs to data labels.
We call them programs because that’s what they are: executable data structures,
and that is what they are called in the genetic algorithm literature, to distinguish them from optimizing bits or strings.
The Brush Program class operates similarly to a sklearn estimator: it has fit and predict methods that are called in during training or inference, respectively.
Types of Programs
There are four fundamental “types” of Brush programs:
Regressors: map inputs to a continous endpoint
Binary Classifiers: map inputs to a binary endpoint, as well as a continuous value in \([0, 1]\)
Multi-class Classifiers: map inputs to a category
Under development
Representors: map inputs to a lower dimensional space.
Under development
Representation
Internally, the programs are represented as syntax trees. We use the tree.hh tree class which gives trees an STL-like feel.
Generation
We generate random programs using Sean Luke’s PTC2 algorithm.
Evaluation
TODO
Visualizing Programs
Programs in Brush are symbolic tree structures, and can be viewed in a few ways:
As a string using
get_model()As a string-like tree using
get_model("tree")As a graph using
graphvizandget_model("dot").
Let’s look at a regresion example.
import pandas as pd
from pybrush import BrushRegressor
# load data
df = pd.read_csv('../examples/datasets/d_enc.csv')
X = df.drop(columns='label')
y = df['label']
# import and make a regressor
est = BrushRegressor(
functions=['SplitBest','Mul','Add','Cos','Exp','Sin'],
max_depth=5,
verbosity=1 # set verbosity==1 to see a progress bar
)
# use like you would a sklearn regressor
est.fit(X,y)
y_pred = est.predict(X)
print('score:', est.score(X,y))
Completed 100% [====================]
score: 0.9441623847546605
You can see the fitness of the final individual by accessing the fitness attribute. Each fitness value corresponds to the objective of same index defined earlier for the BrushRegressor class. By default, it will try to minimize "scorer" and "size", where scorer is a string with a loss function name, set with scorer parameter.
print(est.best_estimator_.fitness)
print(est.objectives)
Fitness(4.757904 74.000000 )
['scorer', 'linear_complexity']
A fitness in Brush is actually more than a tuple. It is a class that has all boolean comparison operators overloaded to allow an ease of use when prototyping with Brush.
It also infers the weight of each objective to automatically handle minimization or maximization objetives.
To see the weights, you can try:
est.best_estimator_.fitness.weights
[-1.0, -1.0]
Other information of the best estimator can also be accessed through its fitness attribute:
print(est.best_estimator_.fitness.size)
print(est.best_estimator_.fitness.complexity)
print(est.best_estimator_.fitness.depth)
28
134628
5
Serialization
Brush let’s you serialize the entire individual, or just the program or fitness it wraps. It uses JSON to serialize the objects, and this is implemented with the get and set states of an object:
estimator_dict = est.best_estimator_.__getstate__()
for k, v in estimator_dict.items():
print(k, v)
fitness {'complexity': 134628, 'crowding_dist': 0.0984368622303009, 'dcounter': 0, 'depth': 5, 'dominated': [0, 15], 'linear_complexity': 74, 'loss': 5.118783950805664, 'loss_v': 4.757904052734375, 'prev_complexity': 134628, 'prev_depth': 5, 'prev_linear_complexity': 74, 'prev_loss': 5.118783950805664, 'prev_loss_v': 4.757904052734375, 'prev_size': 28, 'rank': 1, 'size': 28, 'values': [4.757904052734375, 74.0], 'weights': [-1.0, -1.0], 'wvalues': [-4.757904052734375, -74.0]}
id 227
is_fitted_ False
objectives ['mse', 'linear_complexity']
parent_id [245]
program {'Tree': [{'W': 0.5537276268005371, 'arg_types': ['ArrayF', 'ArrayF'], 'center_op': True, 'feature': '', 'feature_type': 'ArrayF', 'fixed': False, 'is_weighted': False, 'name': 'Add', 'node_type': 'Add', 'prob_change': 1.0, 'ret_type': 'ArrayF', 'sig_dual_hash': 14679000877885575597, 'sig_hash': 14400282083458657357}, {'W': 11.378684043884277, 'arg_types': ['ArrayF'], 'center_op': True, 'feature': '', 'feature_type': 'ArrayF', 'fixed': False, 'is_weighted': True, 'name': 'Exp', 'node_type': 'Exp', 'prob_change': 1.0, 'ret_type': 'ArrayF', 'sig_dual_hash': 13056393536346412951, 'sig_hash': 14128685871577087634}, {'W': 0.9108647704124451, 'arg_types': [], 'center_op': True, 'feature': 'x6', 'feature_type': 'ArrayF', 'fixed': False, 'is_weighted': False, 'name': 'Terminal', 'node_type': 'Terminal', 'prob_change': 0.20750471949577332, 'ret_type': 'ArrayF', 'sig_dual_hash': 7018942542468397869, 'sig_hash': 14162902253047951597}, {'W': 0.7599999904632568, 'arg_types': ['ArrayF', 'ArrayF'], 'center_op': True, 'feature': 'x0', 'feature_type': 'ArrayI', 'fixed': False, 'is_weighted': True, 'name': 'SplitBest', 'node_type': 'SplitBest', 'prob_change': 1.0, 'ret_type': 'ArrayF', 'sig_dual_hash': 14679000877885575597, 'sig_hash': 14400282083458657357}, {'W': 0.8199999928474426, 'arg_types': ['ArrayF', 'ArrayF'], 'center_op': True, 'feature': 'x0', 'feature_type': 'ArrayI', 'fixed': False, 'is_weighted': True, 'name': 'SplitBest', 'node_type': 'SplitBest', 'prob_change': 1.0, 'ret_type': 'ArrayF', 'sig_dual_hash': 14679000877885575597, 'sig_hash': 14400282083458657357}, {'W': 15.890183448791504, 'arg_types': [], 'center_op': True, 'feature': 'constF', 'feature_type': 'ArrayF', 'fixed': False, 'is_weighted': True, 'name': 'Constant', 'node_type': 'Constant', 'prob_change': 0.6167147755622864, 'ret_type': 'ArrayF', 'sig_dual_hash': 7018942542468397869, 'sig_hash': 14162902253047951597}, {'W': 0.17655502259731293, 'arg_types': [], 'center_op': True, 'feature': 'x3', 'feature_type': 'ArrayF', 'fixed': False, 'is_weighted': True, 'name': 'Terminal', 'node_type': 'Terminal', 'prob_change': 0.8625447154045105, 'ret_type': 'ArrayF', 'sig_dual_hash': 7018942542468397869, 'sig_hash': 14162902253047951597}, {'W': 0.14056099951267242, 'arg_types': ['ArrayF'], 'center_op': True, 'feature': '', 'feature_type': 'ArrayF', 'fixed': False, 'is_weighted': False, 'name': 'Exp', 'node_type': 'Exp', 'prob_change': 1.0, 'ret_type': 'ArrayF', 'sig_dual_hash': 13056393536346412951, 'sig_hash': 14128685871577087634}, {'W': 1.7882635593414307, 'arg_types': ['ArrayF'], 'center_op': True, 'feature': '', 'feature_type': 'ArrayF', 'fixed': False, 'is_weighted': True, 'name': 'Sin', 'node_type': 'Sin', 'prob_change': 1.0, 'ret_type': 'ArrayF', 'sig_dual_hash': 13056393536346412951, 'sig_hash': 14128685871577087634}, {'W': 0.8196039795875549, 'arg_types': [], 'center_op': True, 'feature': 'x1', 'feature_type': 'ArrayF', 'fixed': False, 'is_weighted': True, 'name': 'Terminal', 'node_type': 'Terminal', 'prob_change': 0.6729995608329773, 'ret_type': 'ArrayF', 'sig_dual_hash': 7018942542468397869, 'sig_hash': 14162902253047951597}], 'is_fitted_': True}
variation point
With serialization, you can use pickle to save and load just programs or even the entire individual.
import pickle
import os, tempfile
individual_file = os.path.join(tempfile.mkdtemp(), 'individual.json')
with open(individual_file, "wb") as f:
pickle.dump(est.best_estimator_, f)
program_file = os.path.join(tempfile.mkdtemp(), 'program.json')
with open(program_file, "wb") as f:
pickle.dump(est.best_estimator_.program, f)
Then we can load it later with:
with open(individual_file, "rb") as f:
loaded_estimator = pickle.load(f)
print(loaded_estimator.get_model())
Add(11.38*Exp(x6),If(x0>=0.76,If(x0>=0.82,15.89,0.18*x3),Exp(1.79*Sin(0.82*x1))))
String
Now that we have trained a model, est.best_estimator_ contains our symbolic model.
We can view it as a string:
print(est.best_estimator_.get_model())
Add(11.38*Exp(x6),If(x0>=0.76,If(x0>=0.82,15.89,0.18*x3),Exp(1.79*Sin(0.82*x1))))
Quick Little Tree
Or, we can view it as a compact tree:
print(est.best_estimator_.get_model("tree"))
Add
|- 11.38*Exp
| |- x6
|- If(x0>=0.76)
| |- If(x0>=0.82)
| | |- 15.89
| | |- 0.18*x3
| |- Exp
| | |- 1.79*Sin
| | | |- 0.82*x1
GraphViz
If we are feeling fancy 🎩, we can also view it as a graph in dot format. Let’s import graphviz and make a nicer plot.
import graphviz
model = est.best_estimator_.get_model("dot")
graphviz.Source(model)
The model variable is now a little program in the dot language, which we can inspect directly.
print(model)
digraph G {
"141fd3450" [label="Add"];
"141fd3450" -> "141fd2090" [label="11.38"];
"141fd3450" -> "141fcbe00" [label=""];
"141fd2090" [label="Exp"];
"141fd2090" -> "x6" [label=""];
"x6" [label="x6"];
"141fcbe00" [label="x0>=0.76?"];
"141fcbe00" -> "141fe2370" [headlabel="",taillabel="Y"];
"141fcbe00" -> "141ff19f0" [headlabel="",taillabel="N"];
"141fe2370" [label="x0>=0.82?"];
"141fe2370" -> "141fe2420" [headlabel="",taillabel="Y"];
"141fe2370" -> "x3" [headlabel="0.18",taillabel="N"];
"141fe2420" [label="15.89"];
"x3" [label="x3"];
"141ff19f0" [label="Exp"];
"141ff19f0" -> "141fd89e0" [label="1.79"];
"141fd89e0" [label="Sin"];
"141fd89e0" -> "x1" [label="0.82"];
"x1" [label="x1"];
}
Tweaking Graphs
The dot manual has lots of options for tweaking the graphs.
You can do this by manually editing model, but brush also provides a function, get_dot_model(), to which you can pass additional arguments to dot.
For example, let’s view the graph from Left-to-Right:
model = est.best_estimator_.get_dot_model("rankdir=LR;")
graphviz.Source(model)
A classification example
from pybrush import BrushClassifier
from sklearn.preprocessing import StandardScaler
# load data
df = pd.read_csv('../examples/datasets/d_analcatdata_aids.csv')
X = df.drop(columns='target')
y = df['target']
est = BrushClassifier(
functions=['SplitBest', 'And', 'Sin', 'Cos', 'Exp'],
max_gens=500,
max_size=50,
max_depth=15,
objectives=["scorer", "linear_complexity"],
scorer="log",
pop_size=100,
bandit='dynamic_thompson',
verbosity=1
)
est.fit(X,y)
print("Best model:", est.best_estimator_.get_model("tree"))
Completed 100% [====================]
Best model: Logistic
|- -0.23+Sum
| |- Sin
| | |- If(AIDS>=16068.00)
| | | |- 70.58
| | | |- If(Total>=1601948.00)
| | | | |- -1.43
| | | | |- If(AIDS>=258.00)
| | | | | |- 70.58
| | | | | |- Total
Notice that classification problems have a fixed logistic root on their programs. When printing the dot model, Brush will highlight fixed nodes using a light coral color.
est.engine_.search_space.print()
print("Best model:", est.best_estimator_.get_model())
print('score:', est.score(X,y))
print("Best model:", est.best_estimator_.get_model("tree"))
model = est.best_estimator_.get_dot_model()
graphviz.Source(model)
=== Search space ===
terminal_map: {"ArrayB": ["1.00"], "ArrayI": ["Age", "Race", "1.00"], "ArrayF": ["AIDS", "Total", "1.00"]}
terminal_weights: {"ArrayB": [0.27714446], "ArrayI": [0.5782708, 0.85288507, 0.491237], "ArrayF": [0.47180814, 0.6575688, 0.07907883]}
SplitBest node_map[ArrayI][["ArrayI", "ArrayI"]][SplitBest] = 1.00*SplitBest, weight = 0.58001214
OffsetSum node_map[MatrixF][["ArrayF", "ArrayF", "ArrayF", "ArrayF"]][OffsetSum] = 0.00+Sum, weight = 0
Logistic node_map[MatrixF][["ArrayF", "ArrayF", "ArrayF", "ArrayF"]][Logistic] = 1.00*Logistic, weight = 0
Sin node_map[MatrixF][["ArrayF", "ArrayF", "ArrayF", "ArrayF"]][Sin] = 1.00*Sin, weight = 0.9063738
Cos node_map[MatrixF][["ArrayF", "ArrayF", "ArrayF", "ArrayF"]][Cos] = 1.00*Cos, weight = 0.60503954
Exp node_map[MatrixF][["ArrayF", "ArrayF", "ArrayF", "ArrayF"]][Exp] = 1.00*Exp, weight = 0.620445
OffsetSum node_map[MatrixF][["ArrayF", "ArrayF", "ArrayF"]][OffsetSum] = 0.00+Sum, weight = 0
Logistic node_map[MatrixF][["ArrayF", "ArrayF", "ArrayF"]][Logistic] = 1.00*Logistic, weight = 0
Sin node_map[MatrixF][["ArrayF", "ArrayF", "ArrayF"]][Sin] = 1.00*Sin, weight = 0.7408498
Cos node_map[MatrixF][["ArrayF", "ArrayF", "ArrayF"]][Cos] = 1.00*Cos, weight = 0.5123497
Exp node_map[MatrixF][["ArrayF", "ArrayF", "ArrayF"]][Exp] = 1.00*Exp, weight = 0.59719974
OffsetSum node_map[MatrixF][["ArrayF", "ArrayF"]][OffsetSum] = 0.00+Sum, weight = 0
Logistic node_map[MatrixF][["ArrayF", "ArrayF"]][Logistic] = 1.00*Logistic, weight = 0
Sin node_map[MatrixF][["ArrayF", "ArrayF"]][Sin] = 1.00*Sin, weight = 0.58827883
Cos node_map[MatrixF][["ArrayF", "ArrayF"]][Cos] = 1.00*Cos, weight = 0.04645303
Exp node_map[MatrixF][["ArrayF", "ArrayF"]][Exp] = 1.00*Exp, weight = 0.5686126
SplitBest node_map[ArrayF][["ArrayF", "ArrayF"]][SplitBest] = 1.00*SplitBest, weight = 0.35056993
OffsetSum node_map[ArrayF][["ArrayF"]][OffsetSum] = 0.00+Sum, weight = 0
Logistic node_map[ArrayF][["ArrayF"]][Logistic] = 1.00*Logistic, weight = 0
Sin node_map[ArrayF][["ArrayF"]][Sin] = 1.00*Sin, weight = 0.90100724
Cos node_map[ArrayF][["ArrayF"]][Cos] = 1.00*Cos, weight = 0.17278638
Exp node_map[ArrayF][["ArrayF"]][Exp] = 1.00*Exp, weight = 0.43059298
Best model: Logistic(Sum(-0.23,Sin(If(AIDS>=16068.00,70.58,If(Total>=1601948.00,-1.43,If(AIDS>=258.00,70.58,Total))))))
score: 0.82
Best model: Logistic
|- -0.23+Sum
| |- Sin
| | |- If(AIDS>=16068.00)
| | | |- 70.58
| | | |- If(Total>=1601948.00)
| | | | |- -1.43
| | | | |- If(AIDS>=258.00)
| | | | | |- 70.58
| | | | | |- Total
