For example imagine asking "can you go into the other room, find object X, and bring it to me"
It seems to me the intelligence/reasoning ability is more than enough for something like that
And giving models images they seem to be able to recognize objects well to understand what they are looking at
But its not clear to me what is holding this product area back.
Is it a matter of the speed of the update loop or number of iterations it takes the agent being too slow to be practical?
Or does interpreting images/video cause the context/cost to blow up?
But in principle I think physical automation is the next big frontier and its probably coming within our lifetimes.
This will come in the form of models that are baked into silicon. AMD recently acquired Taalas, which is a company that specializes in this stuff. They've shown some really promising proof of concepts that result in a nearly 50x increase in token output speed.
Once we see it come to market, things will begin getting really crazy. Not sure how I'm feeling about that.
It is one thing for an LLM to understand when it needs to run a bash script to collect some system info. It is another thing entirely to expect it to be able to take an image, process it, understand depth/distance, map that delta to some co-ordinate system, then call the exact correct motor functions with the right parameters to cover that distance, ( not accounting for positioning hand to correctly grip and pickup the object of arbitrary shape) grab the item and return home.
Doing this using traditional preset instructions and training data is not impossible, but if you mean a truly dynamic AI driven robot, there is just so much variance we arent at a point where it can be consistently producing realistic/desired results (commercially of course.) though I don't think we are very far from it.
TLDR, agentic AI for programming today is still leaps and bounds behind a consistent realistic AI body. Many many many moving pieces where the hallucinations and variance in modern LLMs largely produces inconsistent results. I am not an expert, this is just from anecdotal experience working with very simple robotics and LLMs.
I've seen many videos of LLMs driving arms and other robots.
DrivingBench even lets them drive a Tesla