deeptab.base_models

class deeptab.base_models.Mambular(feature_information, num_classes=1, config=DefaultMambularConfig(lr=0.0001, lr_patience=10, weight_decay=1e-06, lr_factor=0.1, use_embeddings=False, embedding_activation=torch.nn.Identity, embedding_type='linear', embedding_bias=False, layer_norm_after_embedding=False, d_model=64, plr_lite=False, n_frequencies=48, frequencies_init_scale=0.01, embedding_projection=True, batch_norm=False, layer_norm=False, layer_norm_eps=1e-05, activation=torch.nn.SiLU, cat_encoding='int', n_layers=4, d_conv=4, dilation=1, expand_factor=2, bias=False, dropout=0.0, dt_rank='auto', d_state=128, dt_scale=1.0, dt_init='random', dt_max=0.1, dt_min=0.0001, dt_init_floor=0.0001, norm='RMSNorm', conv_bias=False, AD_weight_decay=True, BC_layer_norm=False, shuffle_embeddings=False, head_layer_sizes=[], head_dropout=0.5, head_skip_layers=False, head_activation=torch.nn.SELU, head_use_batch_norm=False, pooling_method='avg', bidirectional=False, use_learnable_interaction=False, use_cls=False, use_pscan=False, mamba_version='mamba-torch'), **kwargs)

A Mambular model for tabular data, integrating feature embeddings, Mamba transformations, and a configurable architecture for processing categorical and numerical features with pooling and normalization.

Parameters:

• cat_feature_info (dict) – Dictionary containing information about categorical features, including their names and dimensions.

• num_feature_info (dict) – Dictionary containing information about numerical features, including their names and dimensions.

• num_classes (int, optional) – The number of output classes or target dimensions for regression, by default 1.

• config (DefaultMambularConfig, optional) – Configuration object with model hyperparameters such as dropout rates, head layer sizes, Mamba version, and other architectural configurations, by default DefaultMambularConfig().

• **kwargs (dict) – Additional keyword arguments for the BaseModel class.

pooling_method

Pooling method to aggregate features after the Mamba layer.

Type:

str

shuffle_embeddings

Flag indicating if embeddings should be shuffled, as specified in the configuration.

Type:

bool

embedding_layer

Layer for embedding categorical and numerical features.

Type:

EmbeddingLayer

mamba

Mamba-based transformation layer based on the version specified in config.

Type:

Mamba or MambaOriginal

norm_f

Normalization layer for the processed features.

Type:

nn.Module

tabular_head

MLP layer to produce the final prediction based on the output of the Mamba layer.

Type:

MLP

perm

Permutation tensor used for shuffling embeddings, if enabled.

Type:

torch.Tensor, optional

forward(num_features, cat_features)

Perform a forward pass through the model, including embedding, Mamba transformation, pooling, and prediction steps.

forward(*data)

Defines the forward pass of the model.

Parameters:

data (tuple) – Input tuple of tensors of num_features, cat_features, embeddings.

Returns:

The output predictions of the model.

Return type:

Tensor

class deeptab.base_models.MLP(feature_information, num_classes=1, config=DefaultMLPConfig(lr=0.0001, lr_patience=10, weight_decay=1e-06, lr_factor=0.1, use_embeddings=False, embedding_activation=torch.nn.Identity, embedding_type='linear', embedding_bias=False, layer_norm_after_embedding=False, d_model=32, plr_lite=False, n_frequencies=48, frequencies_init_scale=0.01, embedding_projection=True, batch_norm=False, layer_norm=False, layer_norm_eps=1e-05, activation=torch.nn.ReLU, cat_encoding='int', layer_sizes=[256, 128, 32], skip_layers=False, dropout=0.2, use_glu=False, skip_connections=False), **kwargs)

A multi-layer perceptron (MLP) model for tabular data processing, with options for embedding, normalization, skip connections, and customizable activation functions.

Parameters:

• cat_feature_info (dict) – Dictionary containing information about categorical features, including their names and dimensions.

• num_feature_info (dict) – Dictionary containing information about numerical features, including their names and dimensions.

• num_classes (int, optional) – The number of output classes or target dimensions for regression, by default 1.

• config (DefaultMLPConfig, optional) – Configuration object with model hyperparameters such as layer sizes, dropout rates, activation functions, embedding settings, and normalization options, by default DefaultMLPConfig().

• **kwargs (dict) – Additional keyword arguments for the BaseModel class.

layer_sizes

List specifying the number of units in each layer of the MLP.

Type:

list of int

cat_feature_info

Stores categorical feature information.

Type:

dict

num_feature_info

Stores numerical feature information.

Type:

dict

layers

List containing the layers of the MLP, including linear layers, normalization layers, and activations.

Type:

nn.ModuleList

skip_connections

Flag indicating whether skip connections are enabled between layers.

Type:

bool

use_glu

Flag indicating if gated linear units (GLU) should be used as the activation function.

Type:

bool

activation

Activation function applied between layers.

Type:

callable

use_embeddings

Flag indicating if embeddings should be used for categorical and numerical features.

Type:

bool

embedding_layer

Embedding layer for features, used if use_embeddings is enabled.

Type:

EmbeddingLayer, optional

norm_f

Normalization layer applied to the output of the first layer, if specified in the configuration.

Type:

nn.Module, optional

forward(num_features, cat_features)

Perform a forward pass through the model, including embedding (if enabled), linear transformations, activation, normalization, and prediction steps.

forward(*data)

Forward pass of the MLP model.

Parameters:

data (tuple) – Input tuple of tensors of num_features, cat_features, embeddings.

Returns:

Output tensor.

Return type:

torch.Tensor

class deeptab.base_models.ResNet(feature_information, num_classes=1, config=DefaultResNetConfig(lr=0.0001, lr_patience=10, weight_decay=1e-06, lr_factor=0.1, use_embeddings=False, embedding_activation=torch.nn.Identity, embedding_type='linear', embedding_bias=False, layer_norm_after_embedding=False, d_model=32, plr_lite=False, n_frequencies=48, frequencies_init_scale=0.01, embedding_projection=True, batch_norm=False, layer_norm=False, layer_norm_eps=1e-05, activation=torch.nn.SELU, cat_encoding='int', layer_sizes=[256, 128, 32], skip_layers=False, dropout=0.5, norm=False, use_glu=False, skip_connections=True, num_blocks=3, average_embeddings=True), **kwargs)

A ResNet model for tabular data, combining feature embeddings, residual blocks, and customizable architecture for processing categorical and numerical features.

Parameters:

• cat_feature_info (dict) – Dictionary containing information about categorical features, including their names and dimensions.

• num_feature_info (dict) – Dictionary containing information about numerical features, including their names and dimensions.

• num_classes (int, optional) – The number of output classes or target dimensions for regression, by default 1.

• config (DefaultResNetConfig, optional) – Configuration object containing model hyperparameters such as layer sizes, number of residual blocks, dropout rates, activation functions, and normalization settings, by default DefaultResNetConfig().

• **kwargs (dict) – Additional keyword arguments for the BaseModel class.

layer_sizes

List specifying the number of units in each layer of the ResNet.

Type:

list of int

cat_feature_info

Stores categorical feature information.

Type:

dict

num_feature_info

Stores numerical feature information.

Type:

dict

activation

Activation function used in the residual blocks.

Type:

callable

use_embeddings

Flag indicating if embeddings should be used for categorical and numerical features.

Type:

bool

embedding_layer

Embedding layer for features, used if use_embeddings is enabled.

Type:

EmbeddingLayer, optional

initial_layer

Initial linear layer to project input features into the model’s hidden dimension.

Type:

nn.Linear

blocks

List of residual blocks to process the hidden representations.

Type:

nn.ModuleList

output_layer

Output layer that produces the final prediction.

Type:

nn.Linear

forward(num_features, cat_features)

Perform a forward pass through the model, including embedding (if enabled), residual blocks, and prediction steps.

forward(*data)

Forward pass of the ResNet model.

Parameters:

data (tuple) – Input tuple of tensors of num_features, cat_features, embeddings.

Returns:

Output tensor.

Return type:

torch.Tensor

class deeptab.base_models.FTTransformer(feature_information, num_classes=1, config=DefaultFTTransformerConfig(lr=0.0001, lr_patience=10, weight_decay=1e-06, lr_factor=0.1, use_embeddings=False, embedding_activation=torch.nn.Identity, embedding_type='linear', embedding_bias=False, layer_norm_after_embedding=False, d_model=128, plr_lite=False, n_frequencies=48, frequencies_init_scale=0.01, embedding_projection=True, batch_norm=False, layer_norm=False, layer_norm_eps=1e-05, activation=torch.nn.SELU, cat_encoding='int', n_layers=4, n_heads=8, attn_dropout=0.2, ff_dropout=0.1, norm='LayerNorm', transformer_activation=torch.nn.Module, transformer_dim_feedforward=256, norm_first=False, bias=True, head_layer_sizes=[], head_dropout=0.5, head_skip_layers=False, head_activation=torch.nn.SELU, head_use_batch_norm=False, pooling_method='avg', use_cls=False), **kwargs)

A Feature Transformer model for tabular data with categorical and numerical features, using embedding, transformer encoding, and pooling to produce final predictions.

Parameters:

• cat_feature_info (dict) – Dictionary containing information about categorical features, including their names and dimensions.

• num_feature_info (dict) – Dictionary containing information about numerical features, including their names and dimensions.

• num_classes (int, optional) – The number of output classes or target dimensions for regression, by default 1.

• config (DefaultFTTransformerConfig, optional) – Configuration object containing model hyperparameters such as dropout rates, hidden layer sizes, transformer settings, and other architectural configurations, by default DefaultFTTransformerConfig().

• **kwargs (dict) – Additional keyword arguments for the BaseModel class.

pooling_method

The pooling method to aggregate features after transformer encoding.

Type:

str

cat_feature_info

Stores categorical feature information.

Type:

dict

num_feature_info

Stores numerical feature information.

Type:

dict

embedding_layer

Layer for embedding categorical and numerical features.

Type:

EmbeddingLayer

norm_f

Normalization layer for the transformer output.

Type:

nn.Module

encoder

Transformer encoder for sequential processing of embedded features.

Type:

nn.TransformerEncoder

tabular_head

MLPhead layer to produce the final prediction based on the output of the transformer encoder.

Type:

MLPhead

forward(num_features, cat_features)

Perform a forward pass through the model, including embedding, transformer encoding, pooling, and prediction steps.

forward(*data)

Defines the forward pass of the model.

Parameters:

data (tuple) – Input tuple of tensors of num_features, cat_features, embeddings.

Returns:

The output predictions of the model.

Return type:

Tensor

class deeptab.base_models.TabTransformer(feature_information, num_classes=1, config=DefaultTabTransformerConfig(lr=0.0001, lr_patience=10, weight_decay=1e-06, lr_factor=0.1, use_embeddings=False, embedding_activation=torch.nn.Identity, embedding_type='linear', embedding_bias=False, layer_norm_after_embedding=False, d_model=128, plr_lite=False, n_frequencies=48, frequencies_init_scale=0.01, embedding_projection=True, batch_norm=False, layer_norm=False, layer_norm_eps=1e-05, activation=torch.nn.SELU, cat_encoding='int', n_layers=4, n_heads=8, attn_dropout=0.2, ff_dropout=0.1, norm='LayerNorm', transformer_activation=torch.nn.Module, transformer_dim_feedforward=512, norm_first=True, bias=True, head_layer_sizes=[], head_dropout=0.5, head_skip_layers=False, head_activation=torch.nn.SELU, head_use_batch_norm=False, pooling_method='avg'), **kwargs)

A PyTorch model for tasks utilizing the Transformer architecture and various normalization techniques.

Parameters:

• cat_feature_info (dict) – Dictionary containing information about categorical features.

• num_feature_info (dict) – Dictionary containing information about numerical features.

• num_classes (int, optional) – Number of output classes (default is 1).

• config (DefaultFTTransformerConfig, optional) – Configuration object containing default hyperparameters for the model (default is DefaultMambularConfig()).

• **kwargs (dict) – Additional keyword arguments.

lr

Learning rate.

Type:

float

lr_patience

Patience for learning rate scheduler.

Type:

int

weight_decay

Weight decay for optimizer.

Type:

float

lr_factor

Factor by which the learning rate will be reduced.

Type:

float

pooling_method

Method to pool the features.

Type:

str

cat_feature_info

Dictionary containing information about categorical features.

Type:

dict

num_feature_info

Dictionary containing information about numerical features.

Type:

dict

embedding_activation

Activation function for embeddings.

Type:

callable

encoder

stack of N encoder layers

Type:

callable

norm_f

Normalization layer.

Type:

nn.Module

num_embeddings

Module list for numerical feature embeddings.

Type:

nn.ModuleList

cat_embeddings

Module list for categorical feature embeddings.

Type:

nn.ModuleList

tabular_head

Multi-layer perceptron head for tabular data.

Type:

MLPhead

cls_token

Class token parameter.

Type:

nn.Parameter

embedding_norm

Layer normalization applied after embedding if specified.

Type:

nn.Module, optional

forward(*data)

Defines the forward pass of the model.

Parameters:

ata (tuple) – Input tuple of tensors of num_features, cat_features, embeddings.

Returns:

The output predictions of the model.

Return type:

Tensor

class deeptab.base_models.TabulaRNN(feature_information, num_classes=1, config=DefaultTabulaRNNConfig(lr=0.0001, lr_patience=10, weight_decay=1e-06, lr_factor=0.1, use_embeddings=False, embedding_activation=torch.nn.Identity, embedding_type='linear', embedding_bias=False, layer_norm_after_embedding=False, d_model=128, plr_lite=False, n_frequencies=48, frequencies_init_scale=0.01, embedding_projection=True, batch_norm=False, layer_norm=False, layer_norm_eps=1e-05, activation=torch.nn.SELU, cat_encoding='int', model_type='RNN', n_layers=4, rnn_dropout=0.2, norm='RMSNorm', residuals=False, head_layer_sizes=[], head_dropout=0.5, head_skip_layers=False, head_activation=torch.nn.SELU, head_use_batch_norm=False, pooling_method='avg', norm_first=False, bias=True, rnn_activation='relu', dim_feedforward=256, d_conv=4, dilation=1, conv_bias=True), **kwargs)

forward(*data)

Defines the forward pass of the model.

Parameters:

• num_features (Tensor) – Tensor containing the numerical features.

• cat_features (Tensor) – Tensor containing the categorical features.

Returns:

The output predictions of the model.

Return type:

Tensor