diff --git a/docs/Application_guide/en/README.md b/docs/Application_guide/en/README.md index 8c6fc991ab0228cc5caff616c9dd17f5b48b8024..b11d88f6af081b1b07f83c070a9504dde1c4e4d2 100644 --- a/docs/Application_guide/en/README.md +++ b/docs/Application_guide/en/README.md @@ -97,6 +97,8 @@ QuecPython Application Guide is a guide on how to use the commonly used function - [DTU Solution](solutions/DTU/README.md) - [Xiaozhi AI chatbot based on websocket](solutions/xiaozhi_AI/README.md) - [Xiaozhi AI chatbot based on MQTT+UDP](solutions/xiaozhi_AI_mqtt/README.md) +- [AIbox](solutions/AIbox/README.md) +- [ChatGPT AI chatbot](solutions/ChatGPT/README.md) - [Smart Agriculture Central Control Panel](solutions/Agriculture-Control-Panel/README.md) - [Wearable Solution](solutions/Wear/readme.md) - [Electricity Meter](solutions/electricity-meter/README.md) diff --git a/docs/Application_guide/en/media/solutions/ChatGPT/EC800MCNLER06A01M08_OCPU_QPY_TEST0917.zip b/docs/Application_guide/en/media/solutions/ChatGPT/EC800MCNLER06A01M08_OCPU_QPY_TEST0917.zip new file mode 100644 index 0000000000000000000000000000000000000000..c144107b9d6d26b8c639ed921d01aa425945599d Binary files /dev/null and b/docs/Application_guide/en/media/solutions/ChatGPT/EC800MCNLER06A01M08_OCPU_QPY_TEST0917.zip differ diff --git a/docs/Application_guide/en/media/solutions/ChatGPT/connect.png b/docs/Application_guide/en/media/solutions/ChatGPT/connect.png new file mode 100644 index 0000000000000000000000000000000000000000..3f990026c530df6ad72fead3a98ff361d20cfa74 Binary files /dev/null and b/docs/Application_guide/en/media/solutions/ChatGPT/connect.png differ diff --git a/docs/Application_guide/en/media/solutions/ChatGPT/pcb.jpg b/docs/Application_guide/en/media/solutions/ChatGPT/pcb.jpg new file mode 100644 index 0000000000000000000000000000000000000000..8b019a2f0eb081b73ac0ecaca262c005dcfdcbf6 Binary files /dev/null and b/docs/Application_guide/en/media/solutions/ChatGPT/pcb.jpg differ diff --git a/docs/Application_guide/en/media/solutions/ChatGPT/software.png b/docs/Application_guide/en/media/solutions/ChatGPT/software.png new file mode 100644 index 0000000000000000000000000000000000000000..3c76fcea183bb3de18ee60133153254e9a6c92d9 Binary files /dev/null and b/docs/Application_guide/en/media/solutions/ChatGPT/software.png differ diff --git a/docs/Application_guide/en/media/solutions/ChatGPT/software_en.png b/docs/Application_guide/en/media/solutions/ChatGPT/software_en.png new file mode 100644 index 0000000000000000000000000000000000000000..a75c81ed2ccb2851795271244d180445e1e3eab1 Binary files /dev/null and b/docs/Application_guide/en/media/solutions/ChatGPT/software_en.png differ diff --git a/docs/Application_guide/en/media/solutions/ChatGPT/test1.png b/docs/Application_guide/en/media/solutions/ChatGPT/test1.png new file mode 100644 index 0000000000000000000000000000000000000000..079b3659879d46800c7236a65be16de2c4624b81 Binary files /dev/null and b/docs/Application_guide/en/media/solutions/ChatGPT/test1.png differ diff --git a/docs/Application_guide/en/media/solutions/ChatGPT/test2.png b/docs/Application_guide/en/media/solutions/ChatGPT/test2.png new file mode 100644 index 0000000000000000000000000000000000000000..638a397bd45a36ba35125a0135339db8177ce403 Binary files /dev/null and b/docs/Application_guide/en/media/solutions/ChatGPT/test2.png differ diff --git a/docs/Application_guide/en/media/solutions/ChatGPT/test3.png b/docs/Application_guide/en/media/solutions/ChatGPT/test3.png new file mode 100644 index 0000000000000000000000000000000000000000..524a222aa8271abc992bc766b2f9546b6127527b Binary files /dev/null and b/docs/Application_guide/en/media/solutions/ChatGPT/test3.png differ diff --git a/docs/Application_guide/en/media/solutions/ChatGPT/test4.png b/docs/Application_guide/en/media/solutions/ChatGPT/test4.png new file mode 100644 index 0000000000000000000000000000000000000000..a2824011b7a4f0270a464880c215068b2dc57ad7 Binary files /dev/null and b/docs/Application_guide/en/media/solutions/ChatGPT/test4.png differ diff --git a/docs/Application_guide/en/sidebar.yaml b/docs/Application_guide/en/sidebar.yaml index c5159c707e736652cb1836a2bbacc9f223099522..21680098a8f52f7f71a06e460326a5d3fbbfe5c3 100644 --- a/docs/Application_guide/en/sidebar.yaml +++ b/docs/Application_guide/en/sidebar.yaml @@ -295,7 +295,16 @@ items: - label: quick start file: solutions/AIbox/quick_start.md - label: software design - file: solutions/AIbox/software_design.md + file: solutions/AIbox/software_design.md + - label: ChatGPT AI chatbot + file: solutions/ChatGPT/README.md + items: + - label: development resources + file: solutions/ChatGPT/dev_resources.md + - label: quick start + file: solutions/ChatGPT/quick_start.md + - label: software design + file: solutions/ChatGPT/software_design.md - label: EP-D200 Hua Series DTU file: solutions/DTU/README.md items: diff --git a/docs/Application_guide/en/solutions/ChatGPT/README.md b/docs/Application_guide/en/solutions/ChatGPT/README.md new file mode 100644 index 0000000000000000000000000000000000000000..3c132e3cb85e3cafa4f6338e0bd8e8844c81640b --- /dev/null +++ b/docs/Application_guide/en/solutions/ChatGPT/README.md @@ -0,0 +1,16 @@ +# ChatGPT AI chatbot + +The ChatGPT AI chatbot solution has the following features: + +- Support English conversations. +- Supports voice interruption/interruption. +- Support voice/press-button wake-up. +- Uses Python language, facilitating secondary development. + + + +------ + +- [Summary of Development Resources](./dev_resources.md) +- [Quick Start](./quick_start.md) +- [Software Design Explanation](./software_design.md) \ No newline at end of file diff --git a/docs/Application_guide/en/solutions/ChatGPT/dev_resources.md b/docs/Application_guide/en/solutions/ChatGPT/dev_resources.md new file mode 100644 index 0000000000000000000000000000000000000000..e8c6dcad7ce81de709374378b4d077cf54c823f0 --- /dev/null +++ b/docs/Application_guide/en/solutions/ChatGPT/dev_resources.md @@ -0,0 +1,36 @@ +# Summary of Development Resources + +## Hardware Information + +- EC800MCNLE development board (including antenna, Type-C data cable, etc.) + + > - [Click here to purchase the development board kit](https://www.quecmall.com/goods-detail/2c90800b9488359c0195efe6367303b5) + +- Computer (Windows 7, Windows 10 or Windows 11) + +- Any speaker with a power of 2-5W ([Purchase link on Quectel Mall](https://www.quecmall.com/goods-detail/2c90800c9488358b01956aa656680239)) + +- A 3.7V polymer lithium battery + +## Software Information + +- USB driver for QuecPython module: [QuecPython_USB_Driver_Win10_ASR](https://developer.quectel.com/wp-content/uploads/2024/09/Quectel_Windows_USB_DriverA_Customer_V1.1.13.zip) +- [QuecPython firmware](../../media/solutions/ChatGPT/EC800MCNLER06A01M08_OCPU_QPY_TEST0917.zip) + +## Development Tools + +- QPYcom - QuecPython debugging tool + - Version: V3.9.0 + - Download [QPYcom](https://developer.quectel.com/wp-content/uploads/2024/09/QPYcom_V3.9.0.zip) +- VSCode - Code editor + - Download [VSCode](https://code.visualstudio.com/) + +## Experimental Source Code + +- Steps to obtain the source code from the Github repository: + + ```bash + git clone https://github.com/QuecPython/solution-ChatGPT.git + ``` + +- [Download compressed package](https://github.com/QuecPython/solution-ChatGPT/archive/refs/heads/main.zip) \ No newline at end of file diff --git a/docs/Application_guide/en/solutions/ChatGPT/quick_start.md b/docs/Application_guide/en/solutions/ChatGPT/quick_start.md new file mode 100644 index 0000000000000000000000000000000000000000..386bc2f87cbb9391f332540d49c346e3b94a168c --- /dev/null +++ b/docs/Application_guide/en/solutions/ChatGPT/quick_start.md @@ -0,0 +1,70 @@ +# Quick Start + +## Hardware Preparation + +- A Windows computer, preferably with the `Win10` system. +- One set of [EC800MCNLE development board](https://www.quecmall.com/goods-detail/2c90800c9488358b0195efe4c2020409) (including antenna, Type-C data cable, etc.). +- A properly functioning SIM card. +- A speaker with a power of 2-5W. + +## Environment Setup + +- Download and install the EC800M series module driver: [QuecPython_USB_Driver_Win10_ASR](https://developer.quectel.com/wp-content/uploads/2024/09/Quectel_Windows_USB_DriverA_Customer_V1.1.13.zip). +- Download and install [VSCode](https://code.visualstudio.com/). +- Download and unzip the [QPYCom](https://images.quectel.com/python/2022/12/QPYcom_V3.6.0.zip) tool to a suitable location on your computer. +- [Download the firmware package](../../media/solutions/ChatGPT/EC800MCNLER06A01M08_OCPU_QPY_TEST0917.zip). +- Download the [experimental source code](https://github.com/QuecPython/solution-ChatGPT/archive/refs/heads/main.zip). + +## Hardware Connection + +Make hardware connections according to the following diagram: + + + +1. Connect the speaker to the pin headers marked with `SPK+` and `SPK-` as shown in the figure. + +2. Connect the antenna to the antenna connection socket marked with "LTE". +3. Insert the available `SIM` card at the indicated position. +4. Connect the battery at the indicated position. +5. Connect the development board to the computer using a Type-C data cable. + +## Device Development + +### Power On + +After completing the hardware connections, if a COM port containing the words `Quectel USB` appears in the port list of the computer's Device Manager, it indicates a successful power-on. + +​ + +### Firmware Package Burning + +Refer to [this chapter](https://developer.quectel.com/doc/quecpython/Application_guide/en/dev-tools/QPYcom/qpycom-dw.html#Download-Script) to burn the firmware package corresponding to the model into the development board. + +### Script Import and Execution + +1. Refer to [this chapter](https://developer.quectel.com/doc/quecpython/Getting_started/en/4G/first_python.html) to import all files in the `src` folder under the source code directory into the module file system, as shown in the following figure: +2. Refer to [this chapter](https://developer.quectel.com/doc/quecpython/Getting_started/en/4G/first_python.html#Run-Script) to execute the main program file `_main.py`. +3. Refer to [this chapter](https://developer.quectel.com/doc/quecpython/Getting_started/en/4G/first_python.html#Stop-a-Running-Program) to stop the program. + +## Business Debugging + +### Program Startup + +After executing the `_main.py` script, the program starts running. + +### Activate Xiaoyuan + +Shout into the microphone "Xiao Yuan Xiao Yuan" (note that the wake-up word is two "Xiao Yuan", not one. If the two are not connected smoothly, it will not be able to be recognized normally. Additionally, if it cannot be recognized normally, you can try slowing down the speaking speed) +Observe the log, successfully connected to the chatgpt websocket, as follows: + + + +After being awakened, it will shout directly into the microphone, such as "play a music" or "play a children's music". The AI will respond or issue the command "ur start playing music". During the music playback, it will attempt to have a voice conversation, observe whether it can play the music alone and re-reply to the conversation, and also check if the AI can respond normally. An example is as follows: + + + +In addition to the conversation function, ChatGPT AI can also recognize voice intentions and manually control the volume level. + + + + diff --git a/docs/Application_guide/en/solutions/ChatGPT/software_design.md b/docs/Application_guide/en/solutions/ChatGPT/software_design.md new file mode 100644 index 0000000000000000000000000000000000000000..61e93cf67712677d8e22535f0b0a4de2ab5ab93c --- /dev/null +++ b/docs/Application_guide/en/solutions/ChatGPT/software_design.md @@ -0,0 +1,200 @@ +# Software Design Explanation + +## Software Framework + +### Scheme Framework + + + + + +## Code Explanation + +### Initialization of the Application class and management of hardware resources + +Initialized all hardware resources (LEDs, audio, buttons) +The protocol communication, threads, synchronization mechanisms, etc. were initialized, laying the foundation for the subsequent business processes. + +```python +class Application(object): + def __init__(self): + # 充电管理 + self.charge_manager = ChargeManager() + + # 各类LED指示灯 + self.wifi_red_led = Led(33) + self.wifi_green_led = Led(32) + self.power_red_led = Led(39) + self.power_green_led = Led(38) + self.lte_red_led = Led(23) + self.lte_green_led = Led(24) + self.led_power_pin = Pin(Pin.GPIO27, Pin.OUT, Pin.PULL_DISABLE, 0) + # 音频管理 + self.aud = audio.Audio(0) + self.aud.set_pa(29) + self.aud.setVolume(11) + self.player = Player(self.aud, ...) + self.pcm = None + self.g711 = None + self.rec = audio.Record(0) + self.rec.ovkws_set_callback(self.on_keyword_spotting) + + # 按键 + self.wakeup_key = ExtInt(ExtInt.GPIO27, ExtInt.IRQ_FALLING, ExtInt.PULL_PU, self.on_wakeup_key_click, 50) + self.vol_plus = ExtInt(ExtInt.GPIO20, ExtInt.IRQ_FALLING, ExtInt.PULL_PU, self.__set_audio_volume, 50) + self.vol_sub = ExtInt(ExtInt.GPIO47, ExtInt.IRQ_FALLING, ExtInt.PULL_PU, self.__set_audio_volume, 50) + + # 协议与线程 + self.protocol = WebSocketClient() + self.protocol.set_callback(event_handler=self.__on_event) + self.chat_thread = None + self.update_realtime_token_thread = Thread(target=self.__update_realtime_token_thread_handler) + + + # 同步与锁 + self.__semphore = BoundedSemaphore(value=1) + self.event_set = EventSet() + self.lock = Lock() + self.__kws_thread = None + self.__kws_thread_stop_flag = False +``` + +### Audio resource management and G711 callback + +Dynamically switch between PCM objects and G711 codecs to ensure exclusive access to recording/playback resources and accurate parameters. Through locking protection, prevent multi-thread conflicts. + +```python +def reset_pcm_object(self, samplerate=16000, mode=audio.Audio.PCM.WRITEREAD): + with self.lock: + if self.g711: + self.g711.stop_record_v3() + del self.g711 + self.g711 = None + if self.pcm: + self.pcm.close() + del self.pcm + self.pcm = None + self.pcm = audio.Audio.PCM(0, audio.Audio.PCM.MONO, samplerate, mode, audio.Audio.PCM.BLOCK, 25) + if samplerate == 8000: + self.g711 = G711(self.pcm) + self.g711.set_callback_v3(self.__g711_cb) + self.g711.start_record_v3(0, 200) + logger.debug("change pcm object as {} Hz and mode: {}".format(samplerate, mode)) +``` + +### Dialogue logic + +After the work thread is started, it will activate the thread to execute `__chat_process`. Once human voice is detected, the read audio data will be sent. + +```python +class Application(object): + def __chat_process(self): + self.start_vad() + try: + with self.__protocol: + self.power_red_led.on() + self.__protocol.hello() + self.__protocol.wakeword_detected("小智") + is_listen_flag = False + while True: + data = self.audio_manager.opus_read() + if self.__voice_activity_event.is_set(): + # 有人声 + if not is_listen_flag: + self.__protocol.listen("start") + is_listen_flag = True + self.__protocol.udp_send(data) + # logger.debug("send opus data to server") + else: + if is_listen_flag: + self.__protocol.listen("stop") + is_listen_flag = False + if not self.__protocol.is_state_ok(): + break + # logger.debug("read opus data length: {}".format(len(data))) + except Exception as e: + logger.debug("working thread handler got Exception: {}".format(repr(e))) + finally: + self.power_red_led.blink(250, 250) + self.stop_vad() +``` + +### Wake-up word detection and main process control + +Start the wake-up word detection. The configuration parameters are read from the configuration file, and the thread continuously reads the audio stream. +Once the wake-up word is detected, it automatically enters the main chat process. + +```python +def start_kws(self): + logger.debug("start kws...") + self.reset_pcm_object(samplerate=16000) + value = ql_fs.read_json("/usr/config.json") + self.rec.ovkws_start(value["WAKEUP_KEYWORD"], 0.8) + logger.debug("唤醒词：{}".format(value["WAKEUP_KEYWORD"])) + self.__kws_thread_stop_flag = False + self.__kws_thread = Thread(self.__kws_thread_handler) + self.__kws_thread.start(stack_size=16) + +def on_keyword_spotting(self, state): + logger.info("on_keyword_spotting: {}".format(state)) + if state[0] == 0 and state[1] == 1: + # 唤醒词触发 + self.on_wakeup_key_click(None) +``` + +### Main Business Process (Chat) + +Chat process is mutually exclusive to prevent multiple entries. +During the chat process, switch the audio resources, wait for the protocol session to be established, and keep the connection in the main loop. +The auto-recovery of the wake-up word detection function will be carried out after the chat session ends. + +```python +def on_wakeup_key_click(self, args): + rv = self.__semphore.acquire(block=False) + if not rv: + logger.debug("chat is already running.") + return + self.chat_thread = Thread(target=self.chat_process) + self.chat_thread.start(stack_size=64) + +def chat_process(self): + logger.debug("chat processing...") + self.power_green_led.blink(50, 50) + try: + self.stop_kws() + self.reset_pcm_object(samplerate=8000) + with self.protocol: + if not self.event_set.wait(SESSION_CREATED_EVENT, timeout=10, clear=True): + logger.debug("protocol connect failed, get no SESSION_CREATED_EVENT after 10 seconds.") + return + logger.debug("protocol connect successed") + self.power_green_led.on() + while True: + if not self.protocol.is_state_ok(): + break + utime.sleep(1) + except Exception as e: + usys.print_exception(e) + logger.debug("chat process got {}".format(repr(e))) + finally: + logger.debug("chat process thread break out") + self.__semphore.release() + self.start_kws() + self.power_green_led.blink(250, 250) +``` + + + +### Cloud-based object model and OTA + +Initialize the cloud-based object model, register event callbacks, and support features such as OTA (Over-the-Air) updates, parameter distribution, and attribute synchronization. + +```python +def qth_init(self, pk, ps): + qth_init.init() + qth_config.setServer("mqtt://iot-south.acceleronix.io:1883") + qth_config.setProductInfo(pk, ps) + qth_config.setEventCb({...}) + qth_init.start() +``` + diff --git a/docs/Application_guide/en/solutions/README.md b/docs/Application_guide/en/solutions/README.md index 13186def85e552a4322e2f1060659f12db85cf27..de20a88fedc98a603ddb5adb8a39290d9bfb24e2 100644 --- a/docs/Application_guide/en/solutions/README.md +++ b/docs/Application_guide/en/solutions/README.md @@ -6,6 +6,7 @@ To facilitate your application development and accelerate your project implement - [Xiaozhi AI chatbot based on websocket](./xiaozhi_AI/README.md) - [XiaoZhi_AI_chatbot_mqtt](./xiaozhi_AI_mqtt/README.md) - [AIbox](./AIbox/README.md) +- [ChatGPT AI chatbot](./ChatGPT/README.md) - [EP-D200 Hua Series DTU](./DTU/README.md) - [Smart Agriculture Central Control Panel](./Agriculture-Control-Panel/README.md) - [Wearable Solution](./Wear/readme.md) diff --git a/docs/Application_guide/en/solutions/xiaozhi_AI/README.md b/docs/Application_guide/en/solutions/xiaozhi_AI/README.md index f8f69cc4e61c1e2399afd2887daf9f9ed74b19a1..85f4680490876d20b499e0cf13ee77fe89b4d2df 100644 --- a/docs/Application_guide/en/solutions/xiaozhi_AI/README.md +++ b/docs/Application_guide/en/solutions/xiaozhi_AI/README.md @@ -1,6 +1,6 @@ # Xiaozhi AI chatbot based on websocket -This small intelligent AI chatbot solution has the following features: +The Xiaozhi AI chatbot solution has the following features: - Support for sound tone switching. - Support for voice interruption/break. - Support for voice wake-up. diff --git a/docs/Application_guide/en/solutions/xiaozhi_AI_mqtt/quick_start.md b/docs/Application_guide/en/solutions/xiaozhi_AI_mqtt/quick_start.md index f64dbfc1877b78a7552e775101a8ad086c916aa8..801765e246e5d8907d42fde4bea5ef56b0515d40 100644 --- a/docs/Application_guide/en/solutions/xiaozhi_AI_mqtt/quick_start.md +++ b/docs/Application_guide/en/solutions/xiaozhi_AI_mqtt/quick_start.md @@ -52,7 +52,7 @@ Refer to [this chapter](https://developer.quectel.com/doc/quecpython/Application After executing the `_main.py` script, the program starts running. -### Activate Xiaozhid +### Activate Xiaozhi The state shown in the following figure is the waiting-for-wake-up state. The development board will have a red light flashing, and you need to use the voice "Xiaozhi, Xiaozhi" to wake up Xiaozhi AI for voice conversation. diff --git a/docs/Application_guide/zh/README.md b/docs/Application_guide/zh/README.md index eaf29976b0c61db2c154cd88d341b23b77a30d5c..0548c58dfc79ed1a988eb06d6616490bdbac05c0 100644 --- a/docs/Application_guide/zh/README.md +++ b/docs/Application_guide/zh/README.md @@ -87,6 +87,7 @@ QuecPython 开发指南，是对QuecPython常用功能模块如何使用的指 - [基于websocket的小智 AI 聊天机器人](solutions/xiaozhi_AI/README.md) - [基于MQTT+UDP的小智 AI 聊天机器人](solutions/xiaozhi_AI_mqtt/README.md) - [AIbox聊天机器人](solutions/AIbox/README.md) +- [ChatGPT AI 聊天机器人](solutions/ChatGPT/README.md) - [DTU 方案](solutions/DTU/README.md) - [智慧农业中控面板](solutions/Agriculture-Control-Panel/README.md) - [可穿戴解决方案](solutions/Wear/readme.md) diff --git a/docs/Application_guide/zh/media/solutions/ChatGPT/EC800MCNLER06A01M08_OCPU_QPY_TEST0917.zip b/docs/Application_guide/zh/media/solutions/ChatGPT/EC800MCNLER06A01M08_OCPU_QPY_TEST0917.zip new file mode 100644 index 0000000000000000000000000000000000000000..c144107b9d6d26b8c639ed921d01aa425945599d Binary files /dev/null and b/docs/Application_guide/zh/media/solutions/ChatGPT/EC800MCNLER06A01M08_OCPU_QPY_TEST0917.zip differ diff --git a/docs/Application_guide/zh/media/solutions/ChatGPT/connect.png b/docs/Application_guide/zh/media/solutions/ChatGPT/connect.png new file mode 100644 index 0000000000000000000000000000000000000000..3f990026c530df6ad72fead3a98ff361d20cfa74 Binary files /dev/null and b/docs/Application_guide/zh/media/solutions/ChatGPT/connect.png differ diff --git a/docs/Application_guide/zh/media/solutions/ChatGPT/pcb.jpg b/docs/Application_guide/zh/media/solutions/ChatGPT/pcb.jpg new file mode 100644 index 0000000000000000000000000000000000000000..8b019a2f0eb081b73ac0ecaca262c005dcfdcbf6 Binary files /dev/null and b/docs/Application_guide/zh/media/solutions/ChatGPT/pcb.jpg differ diff --git a/docs/Application_guide/zh/media/solutions/ChatGPT/software.png b/docs/Application_guide/zh/media/solutions/ChatGPT/software.png new file mode 100644 index 0000000000000000000000000000000000000000..3c76fcea183bb3de18ee60133153254e9a6c92d9 Binary files /dev/null and b/docs/Application_guide/zh/media/solutions/ChatGPT/software.png differ diff --git a/docs/Application_guide/zh/media/solutions/ChatGPT/test1.png b/docs/Application_guide/zh/media/solutions/ChatGPT/test1.png new file mode 100644 index 0000000000000000000000000000000000000000..079b3659879d46800c7236a65be16de2c4624b81 Binary files /dev/null and b/docs/Application_guide/zh/media/solutions/ChatGPT/test1.png differ diff --git a/docs/Application_guide/zh/media/solutions/ChatGPT/test2.png b/docs/Application_guide/zh/media/solutions/ChatGPT/test2.png new file mode 100644 index 0000000000000000000000000000000000000000..638a397bd45a36ba35125a0135339db8177ce403 Binary files /dev/null and b/docs/Application_guide/zh/media/solutions/ChatGPT/test2.png differ diff --git a/docs/Application_guide/zh/media/solutions/ChatGPT/test3.png b/docs/Application_guide/zh/media/solutions/ChatGPT/test3.png new file mode 100644 index 0000000000000000000000000000000000000000..524a222aa8271abc992bc766b2f9546b6127527b Binary files /dev/null and b/docs/Application_guide/zh/media/solutions/ChatGPT/test3.png differ diff --git a/docs/Application_guide/zh/media/solutions/ChatGPT/test4.png b/docs/Application_guide/zh/media/solutions/ChatGPT/test4.png new file mode 100644 index 0000000000000000000000000000000000000000..a2824011b7a4f0270a464880c215068b2dc57ad7 Binary files /dev/null and b/docs/Application_guide/zh/media/solutions/ChatGPT/test4.png differ diff --git a/docs/Application_guide/zh/sidebar.yaml b/docs/Application_guide/zh/sidebar.yaml index da0130656ce2ffabbd3ebc8c600b7d7180c4fb23..199237e53b05311331b6fdf6d6f024db5e862d73 100644 --- a/docs/Application_guide/zh/sidebar.yaml +++ b/docs/Application_guide/zh/sidebar.yaml @@ -298,7 +298,16 @@ items: - label: 快速上手 file: solutions/AIbox/quick_start.md - label: 软件设计讲解 - file: solutions/AIbox/software_design.md + file: solutions/AIbox/software_design.md + - label: ChatGPT AI 聊天机器人 + file: solutions/ChatGPT/README.md + items: + - label: 开发资源汇总 + file: solutions/ChatGPT/dev_resources.md + - label: 快速上手 + file: solutions/ChatGPT/quick_start.md + - label: 软件设计讲解 + file: solutions/ChatGPT/software_design.md - label: EP-D200 华系列 DTU file: solutions/DTU/README.md items: diff --git a/docs/Application_guide/zh/solutions/ChatGPT/README.md b/docs/Application_guide/zh/solutions/ChatGPT/README.md new file mode 100644 index 0000000000000000000000000000000000000000..956d0b7904d81b7a749313216b7eebb12f737533 --- /dev/null +++ b/docs/Application_guide/zh/solutions/ChatGPT/README.md @@ -0,0 +1,22 @@ +# ChatGPT AI 聊天机器人 + +该ChatGPT 聊天机器人方案具有以下特性： + +- 支持英文对话。 +- 支持语音中断/打断。 +- 支持语音/按键唤醒。 +- 使用 Python 语言，便于二次开发。 + + + + + + + + + +--- + +- [开发资源汇总](./dev_resources.md) +- [快速上手](./quick_start.md) +- [软件设计讲解](./software_design.md) \ No newline at end of file diff --git a/docs/Application_guide/zh/solutions/ChatGPT/dev_resources.md b/docs/Application_guide/zh/solutions/ChatGPT/dev_resources.md new file mode 100644 index 0000000000000000000000000000000000000000..39d8a27df0cb3d18dcecab295e325a11bab5441e --- /dev/null +++ b/docs/Application_guide/zh/solutions/ChatGPT/dev_resources.md @@ -0,0 +1,34 @@ +# 开发资源汇总 + +## 硬件资料 + + - EC800MCNLE 开发板（含天线、Type-C 数据线等） + + > - [点此购买开发板套件](https://www.quecmall.com/goods-detail/2c90800b9488359c0195efe6367303b5) + + - 电脑（Windows 7、Windows 10 或 Windows 11） + + - 任意 2-5W 功率的喇叭即可 ([移远商城购买链接](https://www.quecmall.com/goods-detail/2c90800c9488358b01956aa656680239)) + + - 一块3.7V 聚合物锂电池 + +## 软件资料 + + - QuecPython 模块的 USB 驱动：[QuecPython_USB_Driver_Win10_ASR](https://developer.quectel.com/wp-content/uploads/2024/09/Quectel_Windows_USB_DriverA_Customer_V1.1.13.zip) + - [QuecPython 固件](../../media/solutions/ChatGPT/EC800MCNLER06A01M08_OCPU_QPY_TEST0917.zip) + +## 开发工具 + +- QPYcom - QuecPython 调试工具 + - 版本：V3.9.0 + - 下载 [QPYcom](https://developer.quectel.com/wp-content/uploads/2024/09/QPYcom_V3.9.0.zip) +- VSCode - 代码编辑器 + - 下载 [VSCode](https://code.visualstudio.com/) + +## 实验源码 + +- 从 Github 仓库获取源代码步骤如下： + ```bash + git clone https://github.com/QuecPython/solution-ChatGPT.git + ``` +- [压缩包下载](https://github.com/QuecPython/solution-ChatGPT/archive/refs/heads/main.zip) \ No newline at end of file diff --git a/docs/Application_guide/zh/solutions/ChatGPT/quick_start.md b/docs/Application_guide/zh/solutions/ChatGPT/quick_start.md new file mode 100644 index 0000000000000000000000000000000000000000..4cb599c398bbdb109b8d91a3fcb0aec8b577dc87 --- /dev/null +++ b/docs/Application_guide/zh/solutions/ChatGPT/quick_start.md @@ -0,0 +1,80 @@ +# 快速上手 + +## 硬件准备 + +- Windows 电脑一台，建议 `Win10` 系统。 +- 一套 [EC800MCNLE 开发板](https://www.quecmall.com/goods-detail/2c90800c9488358b0195efe4c2020409) （含天线、Type-C 数据线等）。 + +- 一张可正常使用的 SIM 卡。 +- 一个 2-5W 功率的喇叭。 + +## 环境搭建 + +- 下载并安装 EC800M 系列模组驱动：[QuecPython_USB_Driver_Win10_ASR](https://developer.quectel.com/wp-content/uploads/2024/09/Quectel_Windows_USB_DriverA_Customer_V1.1.13.zip)。 +- 下载并安装 [VSCode](https://code.visualstudio.com/)。 +- 下载并解压 [QPYCom](https://images.quectel.com/python/2022/12/QPYcom_V3.6.0.zip) 工具到电脑的合适位置。 +- [下载固件包](../../media/solutions/ChatGPT/EC800MCNLER06A01M08_OCPU_QPY_TEST0917.zip)。 +- 下载[实验源码](https://github.com/QuecPython/solution-ChatGPT/archive/refs/heads/main.zip)。 + + +## 硬件连接 + +按照下图进行硬件连接： + + + + +1. 将喇叭连接至图中标识有`SPK+`和`SPK-`的排针上。 + +2. 将天线连接至标识有`LTE`字样的天线连接座上。 + +3. 在图示位置插入可用的 ` SIM` 卡。 + +4. 在图示的位置接入电池。 + +5. 使用 Type-C 数据线连接开发板和电脑。 + + + +## 设备开发 + +### 开机 + +完成硬件连接的工作后，电脑设备管理器的端口列表中出现包含 `Quectel USB` 字样的 COM 口，表示开机成功。 + +![comport.png](../../media/solutions/xiaozhi_AI_mqtt/comport.png) + +### 烧录固件包 + +参考[此章节](https://developer.quectel.com/doc/quecpython/Application_guide/zh/dev-tools/QPYcom/qpycom-dw.html#%E4%B8%8B%E8%BD%BD%E5%9B%BA%E4%BB%B6)，烧录对应型号固件包至开发板。 + +### 脚本导入与运行 + +1. 参考[此章节](https://developer.quectel.com/doc/quecpython/Getting_started/zh/4G/first_python.html)，将源码目录下 `src` 文件夹中的所有文件导入到模组文件系统，如下图所示： + +2. 参考[此章节](https://developer.quectel.com/doc/quecpython/Getting_started/zh/4G/first_python.html#执行脚本文件)，执行主程序文件 `_main.py`。 + +3. 参考[此章节](https://developer.quectel.com/doc/quecpython/Getting_started/zh/4G/first_python.html#停止程序运行)，停止程序运行。 + +## 业务调试 + +### 程序启动 + +执行 `_main.py` 脚本后，程序开始运行。 + +### 激活小远 + +对着麦克风喊话"小远小远”(注意唤醒词是两个"小远"，而不是一个，如果两个之间不连贯则无法正常识别。另外如果无法正常识别，可以尝试放慢语速) +观察日志，成功连接 chatgpt websocket，如下: + + + +唤醒后会近距离对着麦克风喊话，比如"play a music"or"play a children's music"，AI 回复 或者 下发 ur 开始播放音乐，在音乐播放过程中，尝试语音对话，观察能否带单音乐并重新回复对话且AI是否能够正常回复，示例如下图： + + + +除对话还可以ChatGPT AI还可以识别语音意图和手动控制音量大小 + + + + diff --git a/docs/Application_guide/zh/solutions/ChatGPT/software_design.md b/docs/Application_guide/zh/solutions/ChatGPT/software_design.md new file mode 100644 index 0000000000000000000000000000000000000000..3244621f6f3966adfb45f35cebdd6adaa08e9bfd --- /dev/null +++ b/docs/Application_guide/zh/solutions/ChatGPT/software_design.md @@ -0,0 +1,205 @@ +# 软件设计讲解 + +## 软件框架 + +### 方案框架 + + + + + + +## 代码讲解 + +### Application类初始化与硬件资源管理 + +初始化了所有硬件资源（LED、音频、按键） + +初始化了协议通信、线程、同步机制等，为后续业务流程打下基础。 + +```python +class Application(object): + def __init__(self): + # 充电管理 + self.charge_manager = ChargeManager() + + # 各类LED指示灯 + self.wifi_red_led = Led(33) + self.wifi_green_led = Led(32) + self.power_red_led = Led(39) + self.power_green_led = Led(38) + self.lte_red_led = Led(23) + self.lte_green_led = Led(24) + self.led_power_pin = Pin(Pin.GPIO27, Pin.OUT, Pin.PULL_DISABLE, 0) + # 音频管理 + self.aud = audio.Audio(0) + self.aud.set_pa(29) + self.aud.setVolume(11) + self.player = Player(self.aud, ...) + self.pcm = None + self.g711 = None + self.rec = audio.Record(0) + self.rec.ovkws_set_callback(self.on_keyword_spotting) + + # 按键 + self.wakeup_key = ExtInt(ExtInt.GPIO27, ExtInt.IRQ_FALLING, ExtInt.PULL_PU, self.on_wakeup_key_click, 50) + self.vol_plus = ExtInt(ExtInt.GPIO20, ExtInt.IRQ_FALLING, ExtInt.PULL_PU, self.__set_audio_volume, 50) + self.vol_sub = ExtInt(ExtInt.GPIO47, ExtInt.IRQ_FALLING, ExtInt.PULL_PU, self.__set_audio_volume, 50) + + # 协议与线程 + self.protocol = WebSocketClient() + self.protocol.set_callback(event_handler=self.__on_event) + self.chat_thread = None + self.update_realtime_token_thread = Thread(target=self.__update_realtime_token_thread_handler) + + + # 同步与锁 + self.__semphore = BoundedSemaphore(value=1) + self.event_set = EventSet() + self.lock = Lock() + self.__kws_thread = None + self.__kws_thread_stop_flag = False +``` + +## 音频资源管理与G711回调 + +动态切换PCM对象和G711编解码，保证录音/播放资源独占和参数正确。通过锁保护，防止多线程冲突。 + +```python +def reset_pcm_object(self, samplerate=16000, mode=audio.Audio.PCM.WRITEREAD): + with self.lock: + if self.g711: + self.g711.stop_record_v3() + del self.g711 + self.g711 = None + if self.pcm: + self.pcm.close() + del self.pcm + self.pcm = None + self.pcm = audio.Audio.PCM(0, audio.Audio.PCM.MONO, samplerate, mode, audio.Audio.PCM.BLOCK, 25) + if samplerate == 8000: + self.g711 = G711(self.pcm) + self.g711.set_callback_v3(self.__g711_cb) + self.g711.start_record_v3(0, 200) + logger.debug("change pcm object as {} Hz and mode: {}".format(samplerate, mode)) +``` + +### 对话逻辑 + +工作线程开启后会启动线程执行`__chat_process`，检测到人声后就会发送读取的音频数据 + +```python +class Application(object): + def __chat_process(self): + self.start_vad() + try: + with self.__protocol: + self.power_red_led.on() + self.__protocol.hello() + self.__protocol.wakeword_detected("小智") + is_listen_flag = False + while True: + data = self.audio_manager.opus_read() + if self.__voice_activity_event.is_set(): + # 有人声 + if not is_listen_flag: + self.__protocol.listen("start") + is_listen_flag = True + self.__protocol.udp_send(data) + # logger.debug("send opus data to server") + else: + if is_listen_flag: + self.__protocol.listen("stop") + is_listen_flag = False + if not self.__protocol.is_state_ok(): + break + # logger.debug("read opus data length: {}".format(len(data))) + except Exception as e: + logger.debug("working thread handler got Exception: {}".format(repr(e))) + finally: + self.power_red_led.blink(250, 250) + self.stop_vad() +``` + +## 唤醒词检测与主流程控制 + +启动唤醒词检测，配置参数从配置文件读取，线程持续读取音频流。 + +唤醒词检测到后自动进入主聊天流程。 + +```python +def start_kws(self): + logger.debug("start kws...") + self.reset_pcm_object(samplerate=16000) + value = ql_fs.read_json("/usr/config.json") + self.rec.ovkws_start(value["WAKEUP_KEYWORD"], 0.8) + logger.debug("唤醒词：{}".format(value["WAKEUP_KEYWORD"])) + self.__kws_thread_stop_flag = False + self.__kws_thread = Thread(self.__kws_thread_handler) + self.__kws_thread.start(stack_size=16) + +def on_keyword_spotting(self, state): + logger.info("on_keyword_spotting: {}".format(state)) + if state[0] == 0 and state[1] == 1: + # 唤醒词触发 + self.on_wakeup_key_click(None) +``` + +## 主业务流程（聊天） + +聊天流程互斥，防止多次进入。 + +聊天流程中切换音频资源，等待协议会话建立，主循环保持连接。 + +聊天结束后自动恢复唤醒词检测。 + +```python +def on_wakeup_key_click(self, args): + rv = self.__semphore.acquire(block=False) + if not rv: + logger.debug("chat is already running.") + return + self.chat_thread = Thread(target=self.chat_process) + self.chat_thread.start(stack_size=64) + +def chat_process(self): + logger.debug("chat processing...") + self.power_green_led.blink(50, 50) + try: + self.stop_kws() + self.reset_pcm_object(samplerate=8000) + with self.protocol: + if not self.event_set.wait(SESSION_CREATED_EVENT, timeout=10, clear=True): + logger.debug("protocol connect failed, get no SESSION_CREATED_EVENT after 10 seconds.") + return + logger.debug("protocol connect successed") + self.power_green_led.on() + while True: + if not self.protocol.is_state_ok(): + break + utime.sleep(1) + except Exception as e: + usys.print_exception(e) + logger.debug("chat process got {}".format(repr(e))) + finally: + logger.debug("chat process thread break out") + self.__semphore.release() + self.start_kws() + self.power_green_led.blink(250, 250) +``` + + + +## 云端物模型与OTA + +初始化云端物模型，注册事件回调，支持OTA、参数下发、属性同步等。 + +```python +def qth_init(self, pk, ps): + qth_init.init() + qth_config.setServer("mqtt://iot-south.acceleronix.io:1883") + qth_config.setProductInfo(pk, ps) + qth_config.setEventCb({...}) + qth_init.start() +``` + diff --git a/docs/Application_guide/zh/solutions/README.md b/docs/Application_guide/zh/solutions/README.md index afd35410caf6bc3e1f056dfaaac3e6b54087e87b..fef977f72b98f8dd0ec6b51431c171e32fdb3a12 100644 --- a/docs/Application_guide/zh/solutions/README.md +++ b/docs/Application_guide/zh/solutions/README.md @@ -6,6 +6,7 @@ - [基于websocket的小智 AI 聊天机器人](./xiaozhi_AI/README.md) - [基于MQTT+UDP的小智AI聊天机器人](./xiaozhi_AI_mqtt/README.md) - [AIbox](./AIbox/README.md) +- [ChatGPT AI 聊天机器人](solutions/ChatGPT/README.md) - [EP-D200 华系列 DTU](./DTU/README.md) - [智慧农业中控面板](./Agriculture-Control-Panel/README.md) - [可穿戴解决方案](./Wear/readme.md)