Skip to content

Chipintelli Tencent TVS Off Line Solution

1. Scheme introduction

1.1 Scheme overview

CI2306 chip specially customized by Chipintelli for offline solution, which integrates offline ASR+WIFI+Bluetooth BLE function, and users can use this chip to quickly develop offline solution; Combined with the advantages of fast offline speech recognition and high recognition rate, combined with cloud NLP online recognition and massive content services, it brings users a new end cloud integration product experience. The solution SDK uses the ci230x released by Chipintelli_ audio_ aiot_ sdk_ Release and ci230x_ wifi_ combo_ sdk_ Release, we will introduce the offline solution based on CI2306 chip+Tencent TVS cloud microenterprise.

1.2 Scheme block diagram

Figure 1-1 Off line Scheme Block Diagram

1.3 Online&offline air conditioning scheme

  • This scheme is an online&offline scheme customized for air conditioners (users can also apply it to other online&offline products as required). The scheme integrates dual MIC+AEC echo cancellation algorithm and blind source separation algorithm, with good noise resistance performance. Users can customize 100+command words; The device supports Bluetooth BLE network and AP network.

2. Rapid development of online&offline scheme

2.1 Development process of Chipintelli CI2306 SDK supported by Tencent Cloud Microenterprise

Figure 2-1
  • Sign in to Tencent Cloud Microenterprise Open Platform

  1. Log in Tencent Cloud Microenterprise Open PlatformClick Login in the upper right corner to register a developer account of Tencent Cloud Microenterprise Open Platform.

  2. After the developer account is registered, use the registered account to log in to the Tencent Cloud Microenterprise Open Platform； You can create device products and cloud skills and debug them. If you create a product and skill release, you need to pass the background audit.

  3. Developer authentication: the developer authentication account refers to the Tencent Cloud Microenterprise Open Platform( https://dingdang.qq.com/open#/)Registered account, and after filling in the enterprise related information, it passed the authentication of Tencent Cloud Microenterprise.

  4. For the specific operation process of developer certification, please refer to Developer Certification Process.

Figure 2-2
  • Note * *: The account registered through the above steps has the highest authority. Please keep your account properly to avoid security risks to the hardware products, apps, device operations or user data you create.
  • The cloud microenterprise open platform supports users to create customized applications according to their own needs. Below is a detailed description of cloud microenterprise application creation and skill configuration
  1. Select the device open platform Click to enter .

Figure 2-3
  1. To create an application, click New Application under the device platform

Figure 2-4
  1. Configure the equipment system and application scenario: select the equipment that is expected to access the application, among which screen equipment includes mobile phone applications, on-board equipment, televisions, watches, screen robots, screen audio, etc., and screen less equipment mainly includes screen less audio, headphones, WeChat official account, screen less robots, etc. Different devices can support different operating systems, which need to be selected according to your actual situation.

Figure 2-5
  1. Fill in the basic information of the application, enter the application name and description, and select the page display method.

Figure 2-6
  1. Fill in the version number in the format of x.x.x.x, such as 1.0.0.0. The version number cannot be modified after it takes effect. The version number is mainly used to update the terminal version, and can also be used to distinguish the created applications.

Figure 2-7
  1. Click Next to view the information related to the created application. After adding the application description, click Finish to complete the application creation and skill configuration.
  • Application Configuration
  1. Basic information configuration: After the application is established, you can view and configure information

Figure 2-8
  1. Version information configuration: Developers can modify skills in [Application Version] - [Skill Configuration]. If you need to launch a formal environment for external users, you must publish it first.

Figure 2-9

  1. For more configurations, please see Tencent Cloud Microenterprise Application and Skill Configuration, which needs to be synchronized with business.

  2. Manufacturer APP access: DMSDK is the abbreviation of Device Management SDK. The DMSDK is accessed by the companion app corresponding to the device. It provides a series of interfaces for the companion app, such as account authorization, device binding, music service authorization, alarm clock management, and voice control. As a bridge between the companion app and the TVS hardware device, it enables users to use the companion app to manage their TVS intelligent devices. The Tencent Cloud Microenterprise Platform provides Android and IOS SDKs. For access, please check ☞Description of manufacturer’s APP access.

2.3 CI230X SDK Access Tencent Cloud Microenterprise Code Framework Description

Figure 2-10

2.4 Tencent TVS cloud access

Figure 2-11

2.5 Enable Tencent TVS

Compilation configuration: Enable Tencent TVS cloud: project ci230x wifi sdk combo CMakeLists.txt

Set (CIAS_IOT_CLOUD_HUAWEI 0) # Huawei Cloud Enable
Set (CIAS_IOT_CLOUD_XIAOMI_ENABLE 0) # Xiaomi Cloud Enable
Set (CIAS_IOT_CLOUD_CI_ENABLE 0) # Enable Qiying Cloud
Set (CIAS_IOT_TENCENT_ENABLE 0) # Tencent IOT Cloud Enable
Set (CIAS_IOT_TVS_ENABLE 1) # Tencent TVS Enable
Set (CIAS_IOT_TUYA_ENABLE 0) # Graffiti Cloud Enable
Set (CIAS_IOT_CLOUD_ALI_ENABLE 0) # Alibaba Cloud Enable

2.6 Call Tencent TVS entry function

project\ci230x-wifi-sdk-combo\app\usr_ ln\usr_ app.c

void creat_ usr_ app_ task(void)
{
...
cias_ tvs_ main();
}

2.7 Modifying Authentication Files

project\ci230x-wifi-sdk-combo\app\cias_ aiot_ wifi\cias_ cloud\cloud_ tvs\tvs_ adapter\tvs_ auth_ manager.h

#define CLOUD_ TVS_ DEFAULT_ PROFILE "QQOPEN,101470979,
9C78976F12BAD3D527D4D5773017D38F,
D53FA920E9791C7BF6B30CF3F3FE50A3,
refreshToken,43733cc6009c4b18,7776000,    feb711004ae411e9b83f6713e1d602f3:db4eee2b600a49b3b7ee3e764513c61c,
8989898911280721"

When testing, you can only modify the value of the last element (898989898911280721) to be authenticated as different authentication files. Each product should be assigned a different authentication file. Otherwise, products with the same authentication file will compete with each other and become offline.

3. API Description of Main Functions

3.1 SDK and Cloud Interaction

3.1.1 Wake up Cloud Identification

Function: after receiving the local wake-up word, call this function to synchronously wake up the cloud to start recognition
Parameter: None
Return value: 0 represents success, others represent failure

int tvs_ api_ impl_ speech_ wakeup()

3.1.2 Upload local audio data

Function: upload PCM audio data compressed by external coding to TVS cloud for cloud identification
Parameter: src_ Addr, upload data address
data_ Len, upload data length
Return value: return the length of successful upload, and the data_ The same len means success

int32_ t outside_ encode_ data_ write(int8_t* src_addr,int32_t data_len)

3.1.3 Actively end cloud identification

Function: call this function after receiving a local vad to actively end cloud recognition
Parameter: None
Return value: None
void tvs_ api_ impl_ speech_ write_ end()

3.1.4 End of cloud identification

Function: callback function after cloud recognition
Parameter: session_ ID, the assigned cloud sessionid identified this time, which can be used for cloud log query
ERROR, the result status of this identification, 0 (TVS_API_AUDIO_PROVIDER_error_NONE) represents success
Return value: None
static void callback_ on_ reader_ stop(int session_id, int error)

3.1.5 Cloud ASR identification result output

Function: After cloud recognition, output the callback function of recognition results
Parameter: asr_ Text, recognition result, unicode character, available for users.
is_ End, 1 means success
Return value: None
void tvs_ callback_ on_ asr(const char* asr_text, bool is_end)

3.2.1 Announcer initialization

Function: Initialize media player adapter, and enable SDK to play network media
Parameter: ad, representing the structure of player operation
Return value: None
void tvs_ init_ mediaplayer_ adapter_ impl(tvs_mediaplayer_adapter *ad)

3.2.2 Start playing

Function: after the cloud sends the media play command, the sdk calls the platform media player to play the streaming media url
Parameter: token, can be ignored
Return value: 0 means success
static int mediaplayer_ adapter_ start_ play(const char* token)

3.2.3 Resume playback

Function: When the SDK exits the busy state, such as when the cloud intelligent voice conversation ends, this interface will be automatically called to continue playing the previous streaming media.
Note: After the recognition of the local command word is successful, the playback will not continue. The playback will enter the paused state, and the active dialogue (such as: continue playing) is required to resume.
Parameter: token, can be ignored
Return value: 0 means success
static int mediaplayer_ adapter_ resume_ play(const char* token)

3.2.4 Stop playing

Function: When the device is playing a streaming media sent from the cloud, the user can say "Stop playing", "Pause playing" and other commands,
Stops the current media playback.
Parameter: token, can be ignored
Return value: 0 means success
static int mediaplayer_ adapter_ stop_ play(const char* token)

3.3 Interaction between SDK and Voice Chip

For the data interaction function between SDK and voice chip, please refer to the CI230X serial port protocol description document

The main APIs are as follows:

3.3.1 Start data receiving task

Function: receive the data of the voice chip, and continuously read the messages in the receiving queue
Parameter: parameter, can be ignored
Return value: None
void cias_ slave_ message_ recv_ deal(void *parameter)

3.3.2 Receive voice chip data

Function: After receiving the voice terminal data, call this function to process the data. The received data includes: control commands sent by the voice terminal, commands to obtain network audio data during playback,
The compressed voice data needs to be uploaded to the cloud,
Parameter: msg_ Buf, data address
msg_ Len, data length
Return value: 0, representing success
int32_ t cias_ slave_ msg_ handle(uint8_t *msg_buf, int32_t msg_len)

3.3.3 Sending voice chip data

Function: The SDK calls this function to send data to the voice chip, including control instructions, network audio data, etc.
Parameter: cmd, data type, see serial port protocol document for specific definitions
Type: DEF by default_ FILL
Len, additional data length
Buf, additional data
Return value: 0, representing success
void cias_ message_ send_ interface(unsigned short cmd, cias_fill_type_t type,
int len, unsigned char *buf)

4. Online&Offline Tencent TVS cloud microenterprise demo use

4.1 DEMO use

4.1.1 Use Chipintelli online&offline development board (note: jumper cap is short circuited according to the figure)

Figure 4-1 Off line Development Board

4.1.2 Distribution network

If you need to reconfigure the WIFI network, please wake up the device (intelligent butler) and call “start the distribution network”. After voice reply, please use Tencent Lianlian applet to start the distribution network. After the distribution network is successfully connected to the cloud, the device will voice back “Network connected”, and the device can be used normally.

4.1.3 Control equipment

The voice control function of the device is edited by the customer. The demo provided initially integrates some voice commands to control the air conditioner and volume. Wake up the device first. After the device replies “I am here”, call for input of control command. The local basic command words provided in the demo are as follows:

Wake up Words Smart Housekeeper
Distribution network command Start distribution network
Command word Raise the temperature Twenty two degrees Maximum wind Maximum volume
Turn on the air conditioner turn up one degree 23 degrees minimum wind minimum volume
Turn off the air conditioner Lower the temperature Twenty four degrees Increase the air volume Increase the wind speed
Turn on cooling turn down one degree 25 degrees reduce air volume reduce wind speed
Turn on heating 16 degrees 26 degrees left and right scavenging start distribution network
Turn on heating 17 degrees 27 degrees Turn off left and right scavenging Exit the distribution network
Turn on dehumidification 18 degrees 28 degrees sweeping up and down
Turn on energy saving 19 degrees 29 degrees Turn off up and down scavenging
Turn off energy saving 20 degrees 30 degrees increase the volume
Turn up the temperature Twenty one degrees Thirty one degrees Decrease the volume

4.1.4 Cloud dialogue

The cloud dialogue capability of the device is supported by Tencent Cloud Microenterprise. First wake up the device, and then call for the desired command after the device replies “I am here”. The commands available for customer testing are as follows, which can be extended by the customer.

Play me a song
I want to listen to children’s songs
I want to listen to pop songs
A pop song
I want to listen to music
Play me a song
Play a song of Qilixiang
Help me open the network song
Play Andy Lau’s song
I want to listen to Li Jian’s song
I want to listen to pure music
I want to listen to pop music
What’s the weather like in Beijing
Smart butler, tell a joke
Smart housekeeper, how many people are there in China
Smart butler, turn the volume to 50%
Smart butler, turn the volume to 100%

If the cloud does not support the input command or voice recognition, it will reply with prompts such as “I did not understand what you said”.