Computer-use AI for family office operations: use cases, risks and controls

Computer-use AI allows an AI agent to operate websites and software through the interface a person would normally use. It can read what is on the screen, navigate to the right place, enter information, and retrieve files without relying on a direct API connection.

That capability is becoming relevant to family offices because some operational work still depends on staff using portals or software manually. Computer use may help with parts of that work, especially when no practical system connection exists.

But, the technology is still uneven. Defined browser tasks are much further along than long workflows across several applications. Direct feeds and APIs remain the better route for structured data, while sensitive actions require tight limits on what an agent can access or do.

The useful question for a family office is where computer use can remove a specific manual step without weakening control.

An API lets two systems exchange information directly. Computer-use AI works through the interface instead.

The agent looks at the software much as a person would. It identifies the relevant page or control, carries out the task, checks what happened, and continues from there.

That opens up software that was never designed to connect easily with other systems.

A manager portal is a simple example. If there is no usable API for retrieving a quarterly report, an agent may be able to log in and download it through the normal interface.

The trade-off is reliability. An API follows a defined technical contract. Computer use has to interpret what it sees, so the outcome is less predictable. Current products increasingly combine visual understanding with browser structure to improve accuracy, but the underlying interaction remains more fragile than a direct system connection.

The strongest cases are narrow tasks inside known systems.

Portal monitoring is one. An agent can check whether a new report or statement has been posted and retrieve it when available.

The same approach may help where staff still enter already validated information into software by hand because the application has no suitable integration.

These jobs are repetitive and easy to verify afterwards. That makes them a more natural fit for current computer-use technology than a long process involving several systems and changing decisions.

Current research still shows weak performance on complex professional workflows. OSWorld 2.0 found that its best tested system fully completed only 20.6% of long-horizon tasks. WindowsWorld found every tested agent below 21% success on multi-application professional work.

A good API or direct feed still has clear advantages when structured information needs to move repeatedly between systems.

Custodian positions and transactions are obvious examples. The office needs consistent data, clear error handling, and a reliable record of what was received. A direct connection is better suited to that job than asking an agent to read the same information from a screen.

Anthropic's own product design follows the same logic. Its Cowork product prefers direct connectors and uses browser interaction when needed because direct integrations are faster and more precise.

Computer use becomes more interesting when that route is unavailable or disproportionate to the task.

Connectivity varies widely across providers. A useful system does not automatically become a good integration target.

Computer use changes the risk profile once an AI system can take action inside software.

A model that only reads a document has limited authority. A model with access to a portal may be able to reach functions that have nothing to do with the task it was given.

Access therefore needs to be restricted at the system level.

An agent retrieving statements from a banking portal should not have payment authority. If the portal presents an unexpected screen or the information does not match what the workflow expects, the task should stop and move to a person.

The research also points to a real security issue around prompt injection. An agent can encounter instructions embedded in webpages or documents while it is operating with privileged access. Anthropic continues to describe browser prompt injection as unresolved, and NIST treats indirect prompt injection as a distinct risk for agent systems.

Financial authority deserves an even harder boundary. Current evidence does not support letting computer-use agents independently execute wires or make other consequential changes. AI may help prepare the work, while approval remains with the people already authorized to give it.

Computer use is one way for a workflow to interact with an application. It does not dictate where the wider operating layer has to run.

A privately managed workflow layer can still control which system the agent is allowed to reach and when a person needs to intervene. The model used for the interface task can then be chosen according to the sensitivity of the work and the deployment options available.

The operating layer holds the process rules and authority boundaries. Computer use handles the specific interaction with the software.

For a family office, that is a much more useful architecture than giving a general-purpose agent broad access and asking it to work out the process for itself. The research consistently points toward bounded agents inside controlled workflows rather than unrestricted autonomy.

Computer-use AI gives family offices another way to deal with software that does not integrate cleanly.

It may prove useful for portal work and other defined interactions that still depend on staff today. The evidence is much weaker for long workflows, desktop applications, and activities involving financial authority.

The technology is moving quickly enough to warrant attention, but the implementation question remains specific to each process.

Where there is a reliable direct connection, use it. Where the work still sits behind an interface, computer use may now be worth testing.