RSS · Spotify · Apple Podcasts · Pocket Casts
He is currently on the faculty job market, and is also looking to help conferences implement new reviewal processes. Some of his ideas include metadata cover sheets to guide readers, and different categories of ML papers with distinct and appropriate criteria for selection (science vs. engineering vs. artifacts). Please feel free to reach out to him!
Highlights from our conversation:
🕸 "Why is sparsity everywhere? This isn't an accident."
🤖 "If I gave you 500 GPUs, could you actually keep those GPUs busy?"
📊 "In general, I think we have a crisis of science in ML."
Below are some highlights and the show notes. As always, please feel free to reach out with feedback, ideas, and questions!
[42:37] The kind of research Jonathan prefers to work on in ML: "So I tend to be a big advocate for doing this more scientific style of research where I ask a question, design an experiment, and make sure that the question you ask has a good answer either way. You can write a paper if the answer is "yes" and you can write a paper where the answer is "no." And everybody will like it either way. It's hard to come by those questions, but I think it's easier to come by a question like that than it is to get an engineering technique to work in a space where you don't know whether it's even possible."
[1:01:49] Building software engineering principles for ML: "My advisor, Michael Carbon, is a programming language researcher, and he sees the world through that point of view. And he keeps referring me to the phenomenon of what's called 'code collapse.' You can look at the Wikipedia article. This was sometime in the seventies when essentially we only had assembly and we only had these relatively low level languages and we didn't know how to build large scale software yet.
Systems just fell over frequently. Systems would just collapse because we didn't know how to make sense out of giant software systems built in this way. We needed new principles and new research on this discipline called software engineering...That's how it feels right now. We don't necessarily know the right principles at the right abstractions. And it's much harder because we're not controlling ones and zeros. We're controlling this weird animal we don't understand that learns in a way we don't understand. But, the same principles may apply. And this should give us some hope that we can get to a better place with these kinds of systems. We can make these into real engineering tools."
Thanks to Luke Cheng for writing drafts of this post and Tessa Hall for editing the podcast.