ICLR (International Conference on Learning Representations)

Our Big Takeaway from ICLR: It's All About Access to Compute

We were in Vienna for ICLR in May 2024, and as always, the scale of the research was incredible. You couldn't walk a single poster aisle without seeing massive transformers, complex generative models, or new self-supervised learning techniques that required immense computational power.

But it was the conversations in the hallways between talks that were most interesting. Beyond the model architectures, the one question that came up constantly was a practical one: "How did you get enough GPU time for that?"

It was clear that access to compute is a huge bottleneck. We talked to many researchers who were tired of maintaining their own hardware and just wanted to spin up a powerful GPU via the cloud for a big training run without a lot of hassle. Moments later, we’d talk to another team that had invested heavily in their own local cluster and wanted to use their own machines for their work.

It was a great reminder that there's no single "right" way to do research. The real need is for flexibility—the ability to use local machines for development and then scale up to the cloud when it's time to really push a model.

As a team that thinks about research infrastructure, it was fascinating to see the community navigating this hybrid future. It’s a complex but really interesting challenge, and it was great to see it being discussed so openly.

A huge thank you to the ICLR organizers for another fantastic conference.

— The CoCalc Team