Driving legacy screens and portals that offer no API
The last mile of an Indian back office is often a screen with nothing behind it you can call — a filing portal, a bank's corporate console, a state department form. A model reads the rendered page or the desktop, decides where to click and what to type, and performs the steps a clerk would, recording the run so somebody can replay what it actually did.
- Effort
- More than a few weeks
- Skill level
- Some technical skill
- Organisation size
- Mid-market
- Value
- Time saved, Cost saved
Tools named for this
- A browser-automation agent that reads the page's accessibility tree rather than only pixels
- A recorded run — screenshots plus the action list — kept as the audit artefact
- A deterministic script for the steps that never change, with the model reserved for the ones that do
What to check before you ship it in India
- Screen recordings of a filing or banking portal capture identifiers, account numbers and full names in the clear, and the recording is a new store of personal data you created and control. Section 8(5) puts the duty of reasonable security safeguards to prevent a personal data breach on the fiduciary, and run artefacts kept for debugging routinely outlive the purpose that justified them.
- Evaluation on real desktops names the two standing weaknesses as locating the right element on screen and knowing the application's own conventions. A portal that renames a field is not a broken selector to this kind of system; it is a slightly harder guess, and it will make one.
Sources
Every claim on this page traces to one of these, on the date it was read.
- OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments · arXiv (Xie, Zhang, Chen, Li, Zhao, Cao, Toh, Cheng, Shin, Lei, Liu, Xu, Zhou, Savarese, Xiong, Zhong, Yu; University of Hong Kong and Salesforce Research) · how it is done · read 2026-09-01
- WebArena: A Realistic Web Environment for Building Autonomous Agents · arXiv (Zhou, Xu, Zhu, Zhou, Lo, Sridhar, Cheng, Ou, Bisk, Fried, Alon, Neubig; Carnegie Mellon University) · how it is done · read 2026-09-01
- The Digital Personal Data Protection Act, 2023 (No. 22 of 2023) — most obligations commence 13 May 2027 under the DPDP Rules 2025 — s.8(5) · Ministry of Electronics and Information Technology · a rule · read 2026-09-01
- WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?