Orylo Computer Use
Let your assistants work with Mac apps and Chrome, with access you approve and control you can stop.
Sections in this article
What it does
Computer Use lets an assigned Assistant inspect supported app interfaces, click controls, enter text, and work with Chrome tabs. It is built into Orylo; you do not need a separate Node or Bun installation. Available actions depend on the target app and the Assistant’s tool support.
Enable it for an Assistant
Open Settings → MCP Servers → Library. Find Computer Use under Built into Orylo, enable it, and assign it to the Assistant you want to use. It is not assigned to every Assistant by default. Start a conversation with that Assistant and name the app and task clearly.
Approve access
When the Assistant requests an app, review the application shown in the approval card. Approval applies to the current session. macOS Accessibility permission is needed for native interaction; screenshots and visual observation may also require Screen Recording permission. Grant these to Orylo in System Settings → Privacy & Security when prompted. For Chrome, also confirm remote-debugging access yourself. If Orylo opens Chrome’s remote-debugging settings, enable the option and complete Chrome’s confirmation.
Preview and end control
While targets are active, use the screen icon beside Composer’s token usage to open previews. Switch between targets, or click a preview title to return to its app or tab. Closing the preview does not stop control. Use End Control to stop that target; stop the conversation to interrupt its current execution. Actions already performed are not undone. If you interact with the controlled app, subsequent automated input pauses until the interaction protection allows it to resume after a fresh observation.
Observe changes
From Orylo 0.1.9, you can ask the Assistant to watch a selected app or browser tab and continue the conversation when a relevant change is detected. Specify the target, the condition, and what to do next. View active observation tasks from Composer, open their targets, or stop individual tasks. Closing the preview does not stop observation. Tasks do not survive an Orylo restart; visual observation may be unavailable for hidden or minimized windows.
Try a clear, bounded request
“In Notes, read this draft and suggest three clearer headings. Do not edit it yet.” Or: “Watch this Chrome tab for the export to finish, then tell me. Do not download or submit anything.” These are example requests, not guarantees that every app or page supports the workflow.
Privacy and troubleshooting
The selected Assistant may receive interface text and requested screenshots or observation images. Your configured Agent or model provider handles those inputs; Computer Use is not a promise of entirely local AI processing. Orylo Backend does not relay these Agent requests. If tools are missing, check the assignment and enabled tools. If access fails, check app approval, macOS permissions, and Chrome’s connection. Review any partial changes before retrying. Removing the MCP assignment revokes that Assistant’s access through Orylo; macOS permissions are managed separately in System Settings.