Question: use vision AI with element position and click

[guillegette]

Hi team,

Sorry for creating an issue about this, please let me know if there is a better place to share thoughts and questions.

I am really enjoying this project but I can already see many instances where the framework is not able to solve the request. Is there a reason why we couldn't use a AI vision model where we give a screenshot of the web page and ask "where is the red button", so we get coordinates back that we can then pass back to the browser and click on it?

Would love to hear your thoughts about this.

[pkiv]

Hey @guillegette! Thanks for sharing. I think this would be a great modifier to the useVision argument (useVisionCoordinates maybe?) We're discussing this in the community Slack here: https://stagehand-dev.slack.com/archives/C07UCP76U8G/p1730392286230649