Sholay
Research into how an AI that operates a Mac handles brand-new tasks versus familiar ones.
- Status
- Research prototype
- Year
- 2026
- Role
- Solo
- Area
- Developer tools
In plain English
AI agents can now use a computer the way a person does, by looking at the screen and clicking. I wanted to know how they cope with a task they have never seen compared with a routine, well-defined one. Sholay is the agent and test bench I built to study that question, with a person able to step in at every stage.
How it works
- The agent takes a screenshot of the Mac and chooses one action at a time, such as a click or keystroke.
- Every action is graded by risk. Risky ones wait for human approval, and dangerous ones never run.
- Open-ended tasks test zero-shot behavior (handling a task with no prior examples or script).
- Predefined, step-by-step flows test familiar tasks, recording cost, accuracy, tokens and time for every run.
- Each run saves screenshots, a transcript and an approvals log for review afterward.
What I did
I designed and built it solo as research for my work. Most of the historical run data was lost, so what I share here is the design and the question.
What I learned
People should see what an agent plans to do before it acts. Enforcing that in code, where the model can't skip it, made the agent both safer to run and easier to study.