#HarmonyOS

Midscene connects to HarmonyOS NEXT devices through HarmonyOS Device Connector (HDC) to automate apps and system interfaces.

This guide covers device connection, model configuration, Playground, and JavaScript SDK integration with @midscene/harmony.

#See it in action

Prompt: Open Settings, find About phone, and view the device information.

View the full report, or explore more Midscene showcases.

#Get started

#Prepare your HarmonyOS device

Before writing scripts, verify that HDC can connect to your device and the device trusts the current computer.

#Install HDC

HDC (HarmonyOS Device Connector) is a command-line tool provided by HarmonyOS for communicating with devices. Installation options:

• Via DevEco Studio (recommended)
• Via HarmonyOS command-line tools standalone installation

Verify HDC is installed:

hdc version

A version number in the output confirms successful installation.

export HDC_HOME=/path/to/hdc/directory

#Enable Developer Mode and verify the device

In your HarmonyOS device settings, go to Developer Options and enable USB Debugging, then connect via USB cable.

Verify the connection:

hdc list targets

A device ID in the output confirms a successful connection:

0123456789ABCDEF

#Launch Playground

Playground is the fastest way to validate the connection and try core capabilities such as aiAct, aiQuery, and aiAssert without writing code. It shares the same core as @midscene/harmony, so anything that works here will behave the same once scripted.

1. Launch the Playground CLI:

npx --yes @midscene/harmony-playground

2. Click the gear button in the Playground window and paste your API Key configuration. See Supported models and setup if you still need a model configuration.

#Use the JavaScript SDK

Once Playground runs successfully, you can switch to reusable JavaScript scripts.

#Configure the model

Set the model configuration through environment variables. For supported models and copyable setup examples, see Supported models and setup.

export MIDSCENE_MODEL_BASE_URL="https://replace-with-your-model-service-url/v1"
export MIDSCENE_MODEL_API_KEY="replace-with-your-api-key"
export MIDSCENE_MODEL_NAME="replace-with-your-model-name"
export MIDSCENE_MODEL_FAMILY="replace-with-your-model-family"

For all configuration options, see Model configuration.

#Install dependencies

npm install @midscene/harmony dotenv --save-dev

yarn add @midscene/harmony dotenv --save-dev

pnpm add @midscene/harmony dotenv --save-dev

bun add @midscene/harmony dotenv --save-dev

deno add npm:@midscene/harmony npm:dotenv --save-dev

#Write a script

The following example opens the Settings app on the device and performs scrolling operations.

import 'dotenv/config'; // load Midscene environment variables from .env if present
import {
  HarmonyAgent,
  HarmonyDevice,
  getConnectedDevices,
} from '@midscene/harmony';

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
Promise.resolve(
  (async () => {
    const devices = await getConnectedDevices();
    const device = new HarmonyDevice(devices[0].deviceId, {});

    const agent = new HarmonyAgent(device, {
      aiActionContext:
        'This is a HarmonyOS device. The system language is Chinese. If any popup appears, dismiss or agree to it.',
    });
    await device.connect();

    // Open Settings app
    await agent.launch('com.huawei.hmos.settings');
    await sleep(2000);

    // Scroll down
    await agent.aiAct('scroll down one screen');

    // Query page content
    const items = await agent.aiQuery(
      'string[], list all visible setting item names',
    );
    console.log('Settings items', items);

    // Assert
    await agent.aiAssert('There is a settings item list on the page');
  })(),
);

#Run the script

npx tsx demo.ts

#View the report

After a successful run, the console outputs Midscene - report file updated: /path/to/report/some_id.html. Open this HTML file in a browser to replay each interaction, query, and assertion.

#Advanced

Use this section to customize device behavior, integrate Midscene into a standalone framework, or troubleshoot HDC issues. See the HarmonyOS section of the API reference for more constructor parameters.

#Extending Midscene on HarmonyOS

Use defineAction() to define custom gestures and pass them via customActions. Midscene appends custom actions to the planner, allowing AI to invoke your domain-specific action names.

import { getMidsceneLocationSchema, z } from '@midscene/core';
import { defineAction } from '@midscene/core/device';
import { HarmonyAgent, HarmonyDevice, getConnectedDevices } from '@midscene/harmony';

const ContinuousClick = defineAction({
  name: 'continuousClick',
  description: 'Click the same target repeatedly',
  paramSchema: z.object({
    locate: getMidsceneLocationSchema(),
    count: z.number().int().positive().describe('How many times to click'),
  }),
  async call(param) {
    const { locate, count } = param;
    console.log('click target center', locate.center);
    console.log('click count', count);
  },
});

const devices = await getConnectedDevices();
const device = new HarmonyDevice(devices[0].deviceId, {});
await device.connect();

const agent = new HarmonyAgent(device, {
  customActions: [ContinuousClick],
});

await agent.aiAct('click the red button five times');

For more details on custom actions and action schemas, see Integrate with Any Interface.

#More

• View all Agent methods: API Reference (Common)
• HarmonyOS-specific parameters and interfaces: API reference (HarmonyOS)
• Use YAML automation scripts and command-line tools.
• Demo projects
  • HarmonyOS JavaScript SDK demo: https://github.com/web-infra-dev/midscene-example/blob/main/harmony/javascript-sdk-demo
  • HarmonyOS + Vitest demo: https://github.com/web-infra-dev/midscene-example/tree/main/harmony/vitest-demo

#FAQ

#Keyboard is not dismissed or the page goes back after typing

Midscene automatically dismisses the keyboard after entering text. By default, HarmonyOS uses the ESC key so the current page is less likely to navigate back. If ESC does not close the keyboard in your app, switch to Back first:

const device = new HarmonyDevice('device-id', {
  keyboardDismissStrategy: 'back-first',
});

If your input field listens for Back and clears or closes in response, disable auto keyboard dismiss:

const device = new HarmonyDevice('device-id', {
  autoDismissKeyboard: false,
});

With auto dismiss disabled, the keyboard will remain visible. You can use aiAct to manually dismiss it, e.g. await agent.aiAct('dismiss the keyboard').

#How to use a custom HDC path?

Set the HDC_HOME environment variable to point to the HDC directory:

export HDC_HOME=/path/to/hdc/directory

Or pass it via the constructor:

const device = new HarmonyDevice('0123456789ABCDEF', {
  hdcPath: '/path/to/hdc',
});