ForkFree

AI agents and automation

agent-desktop

by vercel-labs

agent-desktop is a command-line tool that lets an AI agent see and operate desktop apps on macOS, Windows and Linux through the operating system's accessibility tree. It returns structured JSON listing every clickable element, so the agent can act on it and check the result without matching pixels.

Worth a lookEstablished, with a few things to check (listed below).

View on GitHubProject website

Stars
10.8k
Licence
Apache-2.0
Runs on
Your computer
Setup
Moderate
Language
Rust
Age
256 days

What you can do with it

  • Take a snapshot of an app's window and get a list of clickable elements, each with a reference
  • Click, type, scroll or set values in desktop apps by acting on those references
  • Read and manage windows, the clipboard, notifications and screenshots from an agent script

Make money with it

A freelancer could build and sell agent-driven automations for clients who rely on desktop software that has no API.

An idea, not financial advice. Check the licence before charging for anything built on it.

Ways a business could use it

  • Automate repetitive tasks in desktop software that has no API, such as data entry in older accounting or admin apps
  • Let an AI agent test desktop applications by clicking through them and checking the results

Can you use it commercially?

Free to use, change and sell, including in paid products. Keep the copyright notice.

Not legal advice. Read the project's licence file before building a product on it.

Getting it running

Install it globally with npm (npm install -g agent-desktop), then grant Accessibility permission on macOS or install at-spi2-core on Linux.

Follow the official instructions in the project README.

Things to check

  • Snapshots, screenshots and clipboard reads can contain private desktop content, so keep them out of logs and uploads
  • You need to supply and run your own AI agent, which may mean paid model API costs
  • Gives an agent control of your desktop, so mistakes could change or delete data
  • Linux support is uneven, with some features limited to X11 or GNOME
  • Some commands (key-down, key-up, mouse-down, mouse-up) refuse to run for now

Who's talking about it

Checked 03 Nov 2026. Summary drafted by AI from the project's README.

More in AI agents and automation

Mika wangzan101
Hidden gem

Mika is an Android app that carries out one-sentence requests on your phone by operating other apps and websites for you. It works on a hidden background screen, so you can keep using your phone while it runs.

5.7k stars +1341 a day Apache-2.0
iFixAi ifixai-ai
Worth a look

iFixAi is a command-line tool that tests an AI agent by placing it in a simulation of your business and sending it realistic requests, including ones that push it beyond its role. A separate AI judge grades each reply, so you can see whether the agent behaves as intended.

43k stars +1010 a day Apache-2.0
claude-mem thedotmack
Worth a look

Claude-mem records what an AI coding assistant does during a session, summarises it with AI, and feeds the relevant parts back into later sessions. It is for people who want their coding agent to remember project decisions and history instead of starting from scratch each time.

111.1k stars +962 a day Apache-2.0