Memory optimization in large-scale deep learning is mostly about reducing footprint without sacrificing too much performance.

I usually start with mixed precision training (FP16/BF16) to cut memory usage almost in half. Then use gradient checkpointing, which trades a bit of compute for significantly lower memory by recomputing activations during backprop.

For large models, model/data parallelism (like sharding weights across GPUs) is essential. I also keep batch sizes adaptive and use gradient accumulation to simulate larger batches without exceeding memory limits.

On top of that, techniques like pruning, quantization, and efficient architectures help reduce overall model size, while careful data pipeline handling prevents unnecessary memory overhead.