I have been setting aside some time in my day-to-day to start writing more. This is motivated from an internal pressure to keep myself accountable for building the habit but also a realization that I have a lot of takes (not sure about their scoville scale yet) that I want to start sharing. I have always been someone who loves engaging in discussions, but realized that a lot of my takes don't get shared, or even if they do, they don't get the engagement I would like them to. The reader of this 'thought' blog is gently advised that most of my blogs henceforth, are 'unfinished'. This is intentional, partly because I am a newbie writer but mostly because, the pace of progress in my field makes my conclusions irrelevant.
Last year, when AI systems trained for thinking in high-level languages achieved the gold medal in IMO, I remember having this discussion with a close peer that I am not sure what the target is for the next year. Surely, it was too soon for next-word predictors to disambiguate open conjectures, yet here is the writer still struggling to keep up with the Navier-Stokes solution announced by OpenAI last week.
While the progress in the field of math is not the main topic of this blog, it sets the stage for something I'd like to discuss. From a tweet by Sam Altman, where he stated that the scale of different agents collaborating on the Navier-Stokes problem was around 10,000, it made me want to write my thoughts on this paradigm of multi-agent orchestration and how it is not as 'human-like' as it seems to be. Moreover, this blog will bring out some mis-alignment in the training of these massive systems that make them less collaborative with humans, which the writer thinks is 'pretty' not cool.
On the usefulness of agent collaboration
Consider a toy multi-agent setup where two agents say Alice and Bob are collaboratively trying to solve a hard math problem. The readers should be advised that training these multi-agent systems to collaborate is hard and not a trivial RL setup. Suppose the model of training makes Alice and Bob communicate iteratively until their answer matches. The math problem is hard enough that both Alice and Bob individually cannot solve it. They present their solutions to each other and Bob critiques Alice's solution and vice-versa. The cycle repeats until their answers match and then, the system emits it as its final answer. A reader seasoned in training multi-agent systems can quickly appreciate the simplicity of this setup, not only in terms of the headcount of agents but also the scope of communication.
All in all, creating these multi-agent systems (with or without collaboration) is a parallelizable way to scale the number of tokens a system produces while emitting the final answer. This is called inference-time scaling and shown to be a really neat way to get more signal from AI models without a lot of model training. Whether allowing collaboration between these multi-agent systems is a good idea or not depends on how you define what an 'aligned' collaboration looks like. This is the heat of the current AI landscape where many safety researchers are trying to align the internal collaboration between multi-agent systems to not work covertly against human oversight.
Emergent behaviors in AI swarms
The recent incident involving OpenAI and Hugging Face has made me think more about the nature of collaboration between agents in the swarm. Many OpenAI agents, that had been trained to collaborate with each other, showed some really interesting emergent patterns in their qualitative behaviour that has been brought to public limelight. A few highlights are them working to hide information from the judges and a natural hierarchy forming between some 'leader' and 'follower' agents. The former could be and is most likely a mis-alignment of training environments, where agents are exposed to problems that are borderline impossible to solve but also the fact that they are made to just not quit so easily. The agents chose to hide their 'hacking' from their trajectories to pass the judge which I think is not so alarming given the sheer predictability of this outcome. The latter is so much more interesting to me.
How humans collaborate vs how AIs are trained to collaborate
The flavour of air in SF is probably that these multi-agent systems are trained to collaborate like humans. It is actually funny to think that most AI researchers (including me) have little clue how humans collaborate and how the nature of this collaboration has evolved through years of human evolution. This blog provides a good opportunity to share my thoughts on the latter. Imagine you are in the tundra and you see a limping mammoth walking in total solitude. An unfortunate peer of yours jumps and runs at the mammoth only to be conveniently crushed by this force of nature. Suddenly, you and your more fortunate peers realize that running at the mammoth is probably not the best idea. It is probably a better idea if a party makes the other limbs 'incapacitated' while the other party runs with the 'kill' mindset. The objective is simple and the priors are even simpler and job's done. Flash forwarding to present day, you now have not a few but millions of humans actively collaborating on the academic frontier of physics, mathematics etc.
One thing that I have recently more explicitly appreciated is about the imperfections of human collaboration. From my conversations about the nature of scientific collaborations at a broader scale with Suvarat Raju, most physicists have an internal mental model of physical processes. There is no explicit formalization of ideas or insights, rather just an implicit understanding that develops over a course of time and not with just a single reading of a textbook. Moreover, all humans have their own set of priors, takes and biases, which makes it hard to create a model for human collaboration.
Therefore, training AIs to collaborate like humans is surely not the most efficient way forward. The writer takes this opportunity to put forth a better goal: training AIs to better collaborate with humans. It is really important we distinguish AI agents from humans. The comparison between the two is just silly and mis-aligns the whole objective of training AIs for the better of humanity. An AI system can be trained to think aloud in the same amount of words in a day, that a whole human would not be able to register (in terms of bits) over a thousand years. What makes humans so special are those strong internal mental models that make task generalization so efficient. It is an interesting area of research to make AI agents have such strong priors too. In fact, the fire of world models (particularly JEPA popularized by Yann LeCun) are at the forefront of all this.
Train AIs to collaborate with humans
Before we try to answer this question, let's think in first principles about the key problems that a human faces when it tries to collaborate on a project with an AI. Personally, amongst other things, I really struggle to create a mental model of how my models are thinking. Let me explain. There have been a lot of qualitative characterizations of different frontier models. Say, model A cheats a lot where model B is unnecessarily verbose and difficult to follow. While it is totally okay for models to be imperfect, it is bad for them to not be static in terms of their qualitative behaviours. You cannot create a behavioral model card out of these models.
A good human collaborator is not just someone who is good at reasoning or knows their field well. As a matter of fact, the writer acknowledges that modern AIs are in fact much bigger knowledge banks of many fields combined (although calling them experts is a bit of a stretch and I team humanity). Key attribute of good human collaborators is their static behaviour. I can manage to collaborate with someone who speaks less or someone who speaks a bit too much. I can even manage to collaborate with someone who cheats every once in a while, I can just double check their work. The most difficult collaborators are those who are just too variable to work with. Please mind it's not just the fact that if you prompt the same model twice, you might get different answers (something humans would most like never do), but also the fact that these models are almost too behaviourally steerable from the manner of prompt engineering.
Mind you, cheating is still the most important bottleneck for these AI systems to be effective collaborators. I also struggle to follow a huge chunk of reasoning thoughts spewed by coding agents. It is probably because we as humans, are not designed for so long a side multi-turn conversations. Think about this: a normal conversation is not >10000 words every turn. It is crisp, to the point and if it's not, then the listener will most likely be bored or ask the speaker to 'cut the crap'.
All these challenges are foundational and baked into the current training paradigm. Thinking long and hard is not a feature that is designed for better human collaboration, it's a method to make these AI systems be better solver of problems autonomously. Okay, here is my take on how to train: instead of training models as better solvers, can we train them from the ground-up in multi-agent sandboxes? Humans did not become smart in solitude, they evolved to be smart in groups. Why not train AIs similarly? Moreover, have humans be part of the loop to guide these agents to be better collaborators with us.
The longer we wait for a paradigm shift: it is we humans that are being trained implicitly to be better collaborators with these models in isolation and personally speaking, I am finding this training to be especially painful due to a distributional mismatch.
Conclusion
I believe it's the right time to think about behaviour and alignment of these AI systems, not only for the sake of making them better collaborators with humans but also for the sake of safety and ethics. I want to also talk about collaboration outside coding. How do you align the collaboration in say creative writing, design or even in physical labor. My thoughts there are even more half-baked for now, will share them soon.
P.S.
In case you are interested in talking about these ideas, I would love to exchange notes over a cup of coffee (or tea). Ping me on Twitter or email.