ou want to compress large signal data (that doesn’t fit in memory) into a compact form, while retaining enough information to perform a downstream task (e.g., classification) accurately.

Recommended Approach

1. Variational Autoencoder (VAE) + Classifier

   • Train a VAE to compress data into latent vectors.

   • Add a classification head on the latent.

   • Loss = VAE Loss + Classification Loss.

2. Variational Information Bottleneck (VIB)

   • Directly optimizes for compression vs task utility.

   • Minimizes mutual info between input and latent, while maximizing info between latent and label.

3. Contrastive Learning (e.g., SimCLR)

   • Learn useful compressed embeddings without labels.

   • Fine-tune or train a classifier on top later.

4. Streaming/Online Training

   • Process chunks of data sequentially (not all in memory).

   • Use encoders like CNN/RNN for signal windows.

Tools

• PyTorch Lightning / TensorFlow

• Dataloaders with streaming support

• Latent dimension tuning = compression vs accuracy tradeoff