When training classification models on imbalanced datasets, it’s common to use class weights to prevent the model from favoring the majority class. But it got me thinking: If class weights are influencing the optimization process during training, should validation metrics also reflect those same weights? Or should validation always represent the natural distribution of the(Read More)
I’ve spent the last few months building neural network components from scratch to better understand how deep learning works. So far I’ve implemented dense layers, convolutional layers, activation functions, backpropagation, optimizers like SGD and Adam, and basic training loops without relying on PyTorch or TensorFlow. Here’s a simplified example of one of my training loops:(Read More)
I’m training a model, but the loss stops improving after a few epochs. I’ve tried changing the learning rate and training longer, but the results are still poor. What are the first things you check when a deep learning model doesn’t seem to learn? These are phrased to encourage discussion and experience-sharing, making them well(Read More)
I’m training a model, but the loss stops improving after a few epochs. I’ve tried changing the learning rate and training longer, but the results are still poor.
What are the first things you check when a deep learning model doesn’t seem to learn?
These are phrased to encourage discussion and experience-sharing, making them well suited for Reddit or a community forum.
I’m training a CNN on a relatively small image dataset, and the training accuracy quickly reaches near 100%, but validation accuracy stagnates and then drops. I suspect overfitting is the issue. Here’s a simplified version of my training code in PyTorch: import torchimport torch.nn as nnimport torch.optim as optimfrom torchvision import datasets, transforms # Datasettrain_dataset(Read More)
I’m training a CNN on a relatively small image dataset, and the training accuracy quickly reaches near 100%, but validation accuracy stagnates and then drops. I suspect overfitting is the issue.
Here’s a simplified version of my training code in PyTorch:
import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import datasets, transforms
# Dataset
train_dataset = datasets.ImageFolder('data/train', transform=transforms.ToTensor())
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=True)
# Simple CNN
class SimpleCNN(nn.Module):
def __init__(self):
super(SimpleCNN, self).__init__()
self.conv1 = nn.Conv2d(3, 16, 3, padding=1)
self.pool = nn.MaxPool2d(2, 2)
self.fc1 = nn.Linear(16*32*32, 10)
def forward(self, x):
x = self.pool(torch.relu(self.conv1(x)))
x = x.view(-1, 16*32*32)
x = self.fc1(x)
return x
model = SimpleCNN()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
# Training loop
for epoch in range(10):
for inputs, labels in train_loader:
optimizer.zero_grad()
outputs = model(inputs)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()I’ve read about techniques like data augmentation, dropout, and weight regularization, but I’m not sure how to integrate them effectively.
What strategies or best practices would you recommend for reducing overfitting in small datasets while training CNNs?
With newer models getting larger (especially in LLMs and multimodal setups), memory constraints are becoming a major bottleneck during training and inference. Looking for practical approaches others are using to manage this, such as: Gradient checkpointing vs mixed precision Model sharding or distributed training strategies Efficient data loading and batching Would be useful to understand(Read More)
With newer models getting larger (especially in LLMs and multimodal setups), memory constraints are becoming a major bottleneck during training and inference.
Looking for practical approaches others are using to manage this, such as:
Would be useful to understand what’s working in real-world implementations and where trade-offs are being made.
