From RPA to computer-use agents: what changed

0:00 From RPA to computer-use agents

Imagine giving an assistant a plain-language instruction, like check these forty property listings against my spreadsheet and tell me what is wrong, and watching it open a browser and do exactly that. That is what a computer-use agent does. In this lesson you will learn how these agents actually work, how they differ from the automation you already know, and how to pick tasks where they genuinely help instead of creating new problems.

0:32 The agent loop

Every GUI agent runs the same loop. First it perceives: your code captures a screenshot, or a structured view of the page. Then the model reasons: it reads the goal, the steps so far and the current screen, and picks the next action. Then your code acts: it clicks, types or scrolls on a real or virtual machine. Finally something checks whether it worked. Here is the key idea. The model never touches the computer. It only proposes. Your harness decides whether to carry out each action. Every safety control you will build later hangs on that one fact.

1:15 Why it matters now

Why does this matter now? Because a huge share of business software still has no usable API: supplier portals, government sites, legacy admin panels, marketplace dashboards. For years, that meant people copying and pasting between screens. Computer-use agents make that long tail automatable for the first time, which is a genuine opportunity for operations, marketing and QA teams. But they also introduce new costs and risks, so the teams that win will be the ones that choose tasks carefully rather than automating everything in sight.

1:52 Rails, taxis and teleporters

Here's an analogy that makes the whole idea click. A script is like a train on rails. It's fast, cheap and completely predictable, but the moment a rail is missing, it stops dead. A computer-use agent is more like a taxi driver with a map and good eyesight. It's slower and costs more per trip, but when a road is closed it finds another way. And an API is a teleporter: if one exists, you'd be silly to take the taxi. Keep that picture in mind, because choosing between rails, taxis and teleporters is most of the strategy in this course.

2:36 Simple example: one site vs forty

Let's try a simple example. Imagine you need to check whether the Contact us link works on your own website every morning. There's no API for that, but the page structure never changes. That's a train-on-rails job: a ten-line script handles it perfectly and costs almost nothing. Now imagine checking the same thing across forty different client websites, each with a different layout, some changing every month. Writing and maintaining forty scripts gets painful. That's where an agent, or a hybrid of script plus agent, starts to make sense. Same task, different scale and variety, different answer.

3:18 Four ways to automate software

So how is this different from the automation you already know? An API integration talks to structured endpoints. It is fast, cheap and reliable, but only exists where a vendor built one. A Playwright or Selenium script finds elements by selectors. It is deterministic, but it breaks when the interface changes. Classic RPA records clicks and is friendly for business users, but it is brittle at scale. A computer-use agent looks at the screen and reasons, so it adapts to interfaces it has never seen. The trade-off is that it is slower, costs money for every step, and does not behave identically every time.

4:03 Rule of thumb

That leads to the most useful rule in this course. Use the most deterministic tool that can do the job. If there is an API, use it. If a stable script will do, write the script. Reach for an agent when you face the long tail: many different websites, interfaces that change every month, or steps that need judgment. In practice, the strongest systems are hybrids. Scripts handle the predictable parts, and the agent handles the messy parts in between.

4:38 Worked example: listing checks

Let's make this concrete. A real-estate agency in Dubai lists properties on its own site and on several portals. Every Monday, a coordinator spends hours checking that prices, bedrooms and availability match the master sheet. None of the portals offer an API on the agency's plan. So the agency gives an agent one instruction: for each listing in the sheet, open the portal page, compare the fields, and report mismatches, but do not edit anything. The agent runs in a sandboxed browser with a read-only login where the portal allows it. The coordinator reviews the report and makes the fixes.

5:21 Three design choices

Notice three design choices in that example. It is read-only first, so a mistake costs a bad report, not a wrong price shown to customers. The output is structured, so it can be checked and measured. And a human acts on the findings. Only after the report proves accurate for a few weeks would you consider letting the agent make corrections, and even then behind an approval step. This pattern, observe first, act later, act only with approval, will come back again and again.

5:58 Good fit vs poor fit

Agents today are good at filling varied forms, auditing pages for broken links or missing tags, walking through a signup flow like a QA tester, and pulling information from portals with no export. They still struggle with long tasks where one early mistake poisons everything, with canvas-heavy or unusual interfaces, and with anything needing precise timing. And they should never try to defeat CAPTCHAs or bot protection. Those exist for a reason, and bypassing them can breach terms of service.

6:33 Cost and speed, honestly

Let's talk cost and speed honestly, because this is where many pilots disappoint. A script clicks through a page in a fraction of a second. An agent has to capture a screenshot, send it to a model, wait for reasoning, and then act, for every single step. A task a script finishes in seconds can take an agent minutes, and each step consumes tokens. That's fine when the alternative is a person spending an hour, but it's wasteful when a script would do. So always compare against the realistic alternative, not against doing nothing.

7:14 Measure the baseline first

Before any pilot, measure your baseline. Time a real person doing the task five times. Note how often they make mistakes and how long fixing those mistakes takes. Those numbers become your yardstick. Later you'll track the agent's success rate, the minutes humans spend reviewing its work, and cost per run. If you skip the baseline, you'll never know whether the agent actually helped, and you'll end up arguing from anecdotes instead of evidence.

7:46 Three common mistakes

Before we wrap up, three mistakes I see again and again. First, automating the wrong layer: people build an agent to click through a CRM that has a perfectly good API. Second, unbounded tasks: manage our social accounts isn't a task, it's a job description. An agent needs something narrow and checkable. Third, trusting the agent's own report. When an agent says done, that's a claim, not evidence. You'll learn to verify with a screenshot, a database query or a second check. Avoid those three and you're already ahead of most teams experimenting with agents.

8:27 Try this now

Try this now. List three repetitive browser tasks your team does every week. For each one, answer four questions in a quick table. Is there an API? How often does the interface change? Can the task start read-only? How would we check the result? Circle the task with no API, frequent variety, a read-only version and an easy check. That's your first agent candidate. Time how long it takes a person today, five times, so you'll have a baseline when you build it.

9:03 Recap

To recap. A computer-use agent runs a perceive, reason, act and check loop, and the model only proposes actions. Prefer APIs, then scripts, then agents. Start with narrow, read-only, verifiable tasks, and remember that the harness around the model is what makes it safe to deploy. Your next step: list three repetitive browser tasks in your team, and for each one note whether an API exists and whether it can start read-only. In the next lesson, we look at how an agent actually sees a screen.