Multidimensionnal classification

[SeriousJ55]

I just finished chapter 6 "Fine-tuning for classification" and I'm wondering if there's a way to train a multidimensional classifier.

For instance, I would want to classify emails by classes "spam" and "not spam" but also by unrelated classes like "urgent" and "not urgent".

Of course, it can be done by fine tuning two separate GPTs. But do you know if there's a way to do it with only one GPT?

[rasbt]

Hi there,

that's a good question. Like you suggested, this would typically be a hierarchical classification approach where you'd chain 2 models.

However, you could also approach this as a multi-class problem:

• spam
• not-spam-urgent
• not-spam-not-urgent

For this, you'd just add one more class label.

Or you could frame it as a multi-label classification task:

• spam
• not-spam
• urgent
• not urgent

The difference between multi-class and multi-label is that in multi-class only 1 label is selected whereas in multi-label classification multiple labels can be selected. Implementation-wise, multi-label uses a logistic sigmoid function in the output layer:

Here's a simple example

Standard multi-class

y = torch.tensor([2, 0, ...])

class MulticlassExample(nn.Module):
    def __init__(self, input_size, num_classes):
        super().__init__()
        self.output_layer = nn.Linear(layer_input_size, num_classes)

    def forward(self, x):
        return self.output_layer(x)


# computing the loss
outputs = model(x)
loss = criterion(outputs, y)

Multi-label classification

y = torch.tensor([[1, 0, 1], [0, 1, 1]], dtype=torch.float32)  # binary labels

class MultilabelExample(nn.Module):
    def __init__(self, input_size, num_classes):
        super().__init__()
        self.output_layer = nn.Linear(layer_input_size, num_classes)

    def forward(self, x):
        return torch.sigmoid(self.output_layer(x))

# computing the loss
outputs = model(x)
loss = F.binary_cross_entropy(outputs, y)

[SeriousJ55]

I understand. But why do you add a sigmoid for the multi-label classification?

[rasbt]

For softmax, you make the classes mutually exclusive. I.e., if you have 3 classes and one class as probability 0.5, then the others must have <0.5. For sigmoid, you don't have that restriction or dependence on each other. I.e., you can have multiple classes with >= 0.5. (Usually, you say that the class is true if >= 0.5 and false if < 0.5)