We utilize fastText as our base classification model, which you can learn more about from their documentation here.
Test Set Size (Integer) - This is the ratio of testing/training data. Of the documents you assigned to each class, a portion of each dataset will be separated from the training data and used to test the model afterward. The test set size is set to 0.1, or 10% of your input sets by default.
Rebalance - This feature rebalances the input sets so that there is a more equal distribution of labeled data within each class.
This does result in an unequal testing set distribution, as the test set is separated from the training dataset before rebalancing is applied.
Learning Rate - The learning rate of an algorithm indicates how much the model changes after each example sentence is processed. We can both increase and decrease the learning rate of an algorithm. A learning rate of 0 means that there is no change in learning, or the rate of change is just 0, so the model doesn’t change at all. The usual learning rate is 0.1 to 1.
Learning rates can also be optimized automatically. There are several available optimizers publicly available for learning rate optimization. We currently use Adam as our optimizer, an adaptive learning rate optimizer specifically trained for deep neural networks.
Dimensions - The pre-trained word vectors we distribute have a default dimension size, or dimensionality, of 100. This default parameter is different from the default used by the standard fastText base model (which uses 300 dimensions). To increase or decrease the vector dimensionality, adjust this parameter accordingly.
What is dimensionality?
Dimensionality is the size of the word vectors when they are abstracted into what we refer to as “vector space”, which consists of many hundreds or thousands of planes of dimensionality. Dimensionality Reduction (DR) algorithms transform a set of N high-dimensional input data points with M dimensions into an output dataset with the same number of data points but a reduced number of m dimensions: m < M. This technique is used to generate tractable input datasets for classification models, and also aids in visual data analysis application, such as the build-out of t-SNE, UMAP, and PCA plots.
Window Size (Int, Optional) - The maximum distance between a sentence's current and predicted word. This is also the size of the context window.
Epochs - An epoch is the number of times a model sees a phrase or an example input from the training corpora.
Min Word Count (Int, Optional) - The model ignores all words with a total frequency lower than this value.
Min Label Count - The model ignores all labels with a total frequency lower than this value.
This is particularly important if you are using a multi-class label, where your training dataset may only have a handful of examples for a given label. If this is the case, you’ll likely want to consider either (1) gathering more examples for those labels to train on or (2) removing this label from your model.
Min Character NGram Length (Int, Optional) - Minimum length of character n-grams to be used for training word representations.
Max Character NGram Length (Int, Optional) - Maximum length of character ngrams to be used for training word representations. Set max ngram length to be lesser than min ngram length to avoid character ngrams being used.
Max Word NGram Length (Int, Optional) - Maximum length of word n-grams to be used for unigram, bigram, trigram, etc. word handling.
Negatives Sampled (Int, Optional) - If > 0, negative sampling will be used, where the int value for this parameter specifies how many “noise words” should be drawn (usually between 5 and 20). If set to 0, no negative sampling is used.
Loss Function - Loss functions play an important role in any model. There are too many unknowns when calculating the perfect weights for a neural network, which is why the problem of learning is cast as an optimization problem and we utilize loss function algorithms to navigate the space of possible sets of weights the model may use to make good or “good enough” predictions.
Loss functions define an objective which the performance of the model is evaluated against and the parameters learned by the model are determined by minimizing a chosen loss function. In short, loss functions define what a good prediction is and isn’t.
By default, we use the softmax algorithm as our loss function. However, users can also select from three additional algorithms as they choose. We will provide additional documentation on these algorithms by request.*
Available Loss Functions
Softmax (default)
NS - Skipgram negative samples, or SGNS
HS - Skipgram hierarchical softmax
OvA - “One-vs-All”, also referred to as “One-vs-Rest” or OvR
If interested in using other loss functions not currently provided through Signal Curate, we suggest reaching out to your Certara.AI support team for additional guidance.
Buckets (Int, Optional) - Character ngrams are hashed into a fixed number of buckets, to limit the memory usage of the model. This option specifies the number of buckets used by the model. The default value of 2000000 consumes as much memory as having 2000000 more in-vocabulary words in your model.
Learning Rate Update Rate - As mentioned earlier, learning rates can be optimized automatically. There are several available optimizers publicly available for learning rate optimization. We currently use Adam as our optimizer, an adaptive learning rate optimizer that utilizes what’s known as an update rule to speed up optimization performance in terms of speed of training. This update rule is derived from model weights and step size.
If you plan to tinker with this update rate, we suggest connecting with Certara.AI's data scientists to best evaluate different learning rate optimizers for your specific use case.
Sampling Threshold - The threshold for configuring which higher-frequency words are randomly down-sampled, a useful range is (0, 1e-5).