huashu-mac-use: Agents That Drive Mac Apps

Most desktop automation dies at the same wall: the app you need has no API, no command line, and no intention of getting either. huashu-mac-use is a macOS Agent Skill that takes the wall as given. Install it and any agent that can read a SKILL.md file gets a way to drive native Mac apps through whatever channel each one happens to expose, then photograph the result so you can check the work rather than trust it. The repository landed on 6 September 2026 and had 218 stars four days later.

What it actually is, and what it refuses to be

huashu-mac-use is a skill, not an application. There's no window and no menu bar item; you install it into an agent's skills directory and then talk to the agent normally. The repository is small enough to read in an afternoon: a SKILL.md, five reference documents, a scripts/ directory and a Swift core that compiles on first use, with Swift accounting for just over half the code.

The design principle sits at the top of the README as a slogan the rest of the project keeps: read whatever you want, write without interrupting, leave evidence at every step. That third clause is the unusual one, and it drives more of the architecture than the first two.

Four control planes, tried cheapest first

Rather than treating every app as a picture to click on, huashu-mac-use probes first and picks a layer. The gap between layers is measured in orders of magnitude, not percentages.

At the top sits anything structured the app already offers: its own CLI, an AppleScript dictionary, a URL scheme, or a local debugging port speaking CDP or JSON-RPC. That's the default starting point because it costs nothing: no focus stolen, no Space switched, several agents able to work without colliding. Below it, the accessibility tree, reached through mac ax and mac axset, which the skill only counts as working once a write lands and the app's own state visibly changes. Below that, window coordinates: mac see produces a screenshot, mac op clicks the pixel positions you name, and the Swift core does the coordinate conversion so the agent never performs arithmetic on screen geometry. Pixels are the floor for control and the ceiling for proof.

The probe step is where the useful specifics live. Running probe.sh against an app reports its bundle, architecture, URL scheme, AppleScript dictionary, any local port it's listening on, how many editable controls sit in its accessibility tree, and its version. The README calls this step zero, and describes the mistake that produced the rule: one AI desktop client had a complete Chromium engine buried under Contents/Helpers/, with the top-level Frameworks/ directory empty. It got classified as native on day one and driven by coordinate clicking for two days before anyone noticed the debugging port. The fix was to read the process table instead, since a Helper or crashpad_handler child process gives Chromium away, and mac open <app> --cdp 9333 now restarts the app with the port open and waits for it.

"The tool returned success" is not evidence

This is the part worth stealing even if you never install the skill.

huashu-mac-use defines a ladder of proof, weakest to strongest: the tool returned without error, then the value read back, then the text became visible in a screenshot, then the app's own state indicator changed (a send button going from grey to lit, a placeholder disappearing), and finally a side effect landed somewhere else entirely, such as a task appearing in a list or a file written to disk. The project's flat claim is that the first three levels all fool people regularly.

So every action reports a verdict alongside its result: effect=confirmed, partial, suspected_noop, or unverifiable. The skill is explicit that suspected_noop doesn't mean failure; it means go back and look again. Anyone who has watched an agent report filling in a form that stayed empty knows why that belongs in the tooling rather than in judgement.

Reads take nothing; writes ask the machine for permission

Screenshots, window listing, accessibility queries and CDP calls are all reads, and none of them activate an app, switch a Space or move your cursor. Window-level capture works even when the window is fully covered by something else, which is what makes silent background evidence possible in the first place.

Writes climb a ladder instead. The first attempt posts keyboard and mouse events straight to the target process ID, stealing no focus and working through occlusion, and a pixel diff then checks whether anything happened. Only when that comes back inconclusive does the skill consider borrowing focus, and only after passing gates: the frontmost window has to be what you think it is, terminal-style windows need an explicit force flag, a machine-wide lock stops two agents grabbing focus at once, and if you touched the keyboard in the last two seconds it waits up to fifteen for you to stop rather than snatching the cursor mid-sentence. A --dry flag previews every verdict without executing anything.

Worth flagging: the English summary in the README counts four gates and the Chinese SKILL.md describes three. The behaviours listed overlap but the tally doesn't, so read the skill file rather than the summary if the exact sequence matters to you.

While a write runs, a frame pulses at the corners of the screen so you know the agent is acting. It's transparent to the mouse, takes no focus, follows you across Spaces, and is invisible to screen capture, so it never contaminates the evidence it announces.

The whole design exists because one human and one agent are sharing a single keyboard. Take that constraint away and most of the machinery becomes unnecessary, which is the argument for giving the agent its own machine: Zentor is a hosted cloud AI computer with no human elbowing it for the cursor, so nothing has to wait fifteen seconds for you to stop typing.

Where it sits against pure-vision computer use

huashu-mac-use frames itself against computer use that works purely from screenshots and coordinates, and the ideas it puts forward are architectural rather than model capabilities: probe before acting, keep reads out of the focus system, refuse to treat a return value as proof. That's why it ships as a markdown file plus a Swift binary instead of a new model. It also means the approach survives being moved onto other hardware, including a hosted machine like Zentor where the agent isn't sharing a screen with anyone.

The README ends with a line most projects would cut: model-native computer use will probably overtake this within months, and until then here is something that works today. Believe the author on that. What survives the overtaking is the evidence ladder, not the clicking.

The demo the repository leads with is a Blender one: four AI-generated photographs of the same ship from different angles, turned into a parametric model and re-rendered from matching camera positions across eight iterations, with previews landing in five to thirty seconds each. It's a demo, not a benchmark, and the point it proves is narrower than "AI makes 3D models": the agent reached Blender through Blender's own scripting interface, layer zero, and used screenshots only to check its work.

The permission that trips everybody

Grant Screen Recording and Accessibility to the terminal application you run the agent in, not to the script and not to the agent binary. iTerm, Terminal, Cursor, whatever you launched from. Switch terminals and you grant them again from scratch. This catches people constantly, and it's the same class of problem covered in our review of third-party agent skill permissions: the thing macOS is authorising is rarely the thing you think you're authorising.

Two prerequisites are easy to miss: Xcode Command Line Tools, because the Swift core compiles on first run in about twenty seconds, and Node.js for the CDP path into Chromium-based apps. The project reports testing on macOS 26 and expects macOS 14 and later to work, on Apple Silicon or Intel.

Then there's the Spaces rule, which produces a real error string rather than a vague failure. Agents usually run in a full-screen terminal on one desktop while the target app lives on another. Reads cross that boundary fine. Coordinate writes do not: the core refuses with exit code 2 and refused: cross-space, and the documented response is to switch to the CDP path or ask the user to move the window, never to switch Spaces on your own.

What you're actually installing

Five commits. Two contributors, one of which is Claude. Zero releases and zero tags, which means npx skills add alchaincyf/huashu-mac-use pulls whatever is on master at the moment you run it. The licence is MIT, plainly stated in the repository and on the badge. For a project that asks for Accessibility and Screen Recording, reading the Swift core before granting either is time well spent, and at this size that's feasible.

One thing to know before you install: the documentation is Chinese-first. The SKILL.md an agent reads is entirely in Chinese, as are all five reference documents covering control planes, permissions and failure codes, per-app profiles, evidence conventions and a log of what broke during development. Those per-app profiles cover apps popular in China, Blender aside. Agents read Chinese without complaint and the skill's output, which is screenshots, clicks and rendered frames, carries no language at all, so this is a documentation barrier rather than a functional one. It's still a barrier if you want to audit what you're running, and you should want that with these permissions.

Which runtimes work is the loosest claim in the README, ranging from a named list (Claude Code, Codex, Kimi Code, Cursor, OpenClaw) up to "50+ runtimes" in the English summary. The honest version is the specification: anything that reads a SKILL.md can load it, the same portability that lets a good skills library move between agents. Whether it behaves identically on each is untested, and the repository is four days old. The same portability argument applies to the macOS harness browser-use published, which solves an adjacent problem with a local daemon instead of a skill file.

One structural limit is worth naming: none of this leaves the Mac it's installed on. The evidence trail, the focus lock and the screen grants belong to one physical machine, so an overnight render ends when that lid closes. Pairing it with Zentor is the practical split: a hosted cloud AI computer holds the always-on half of the work while huashu-mac-use handles what genuinely needs a local screen. Zentor's trial is three days and 1,000 credits, then $20 a month for 1,000 credits.

FAQ

Can an AI agent control macOS apps?

Yes, and huashu-mac-use is one way to do it without an API. It tries the app's own structured interfaces first, falls back through the accessibility tree to window coordinates, and uses raw pixels mainly to verify that something actually happened. What it can't do is bypass macOS security: Screen Recording and Accessibility have to be granted to the terminal application the agent runs in.

Is huashu-mac-use free?

Yes. It's MIT licensed, published by 花叔 (Huashu), and installable with npx skills add alchaincyf/huashu-mac-use. There is no paid tier in the repository. You will need Xcode Command Line Tools to compile the Swift core, and Node.js for the Chromium path.

What is the difference between this and computer use?

Screenshot-driven computer use looks at pixels and clicks coordinates for every app. huashu-mac-use treats that as its third-best option and probes for something structured first, which keeps most work off the focus system entirely. It also labels each action confirmed, partial, suspected_noop or unverifiable instead of reporting plain success.

Does it need Chinese to use?

Not to operate it. You prompt your agent in whatever language you already use, and the outputs are screenshots and app actions. The skill file and reference documentation are written in Chinese, so auditing the source or reading the failure-code reference does require it, or a translation pass, as of September 2026.

Can it run while I'm using my Mac?

That's the design goal. Reads never take focus, and writes post events straight to the target process before considering anything more intrusive. Touch the keyboard within two seconds and it waits, up to fifteen, rather than interrupting. Work that shouldn't have to negotiate for your cursor at all is the case for handing it to a hosted machine such as Zentor instead.