Coding Challenge 188: Voice Chatbot
From The Coding Train ยท ยท 39 min

The player loads from YouTube only when you click. It uses YouTube's privacy-enhanced mode.
If the video does not play, the owner may have turned off embedding. Watch it on YouTube instead.
Description
In this coding challenge, I build a conversational voice chatbot entirely in the browser with p5.js. I combine three pieces: speech-to-text with OpenAI's Whisper model, text-to-speech with Kokoro TTS, and a "brain" for the bot. I also explore the transformers.js pipeline API and the Web Audio API. For the bot's brain, I start with a simple ELIZA-style therapist, then incorporate a RiveScript number-guessing game, and finally a local LLM. Code: https://thecodingtrain.com/challenges/188-voice-chatbot
๐ Watch this video ad-free on Nebula https://nebula.tv/videos/codingtrain-coding-challenge-188-voice-chatbot
p5.js Web Editor Sketches: ๐น๏ธ LLM Chatbot: https://editor.p5js.org/codingtrain/sketches/RHhT9I4Nm ๐น๏ธ Number Guessing Bot: https://editor.p5js.org/codingtrain/sketches/AJw7zMN9q ๐น๏ธ Therapy Bot: https://editor.p5js.org/codingtrain/sketches/37LFEPUVV ๐น๏ธ Model Loading Bars: https://editor.p5js.org/codingtrain/sketches/E9Ob3x8eJ ๐น๏ธ Waveform of Recording: https://editor.p5js.org/codingtrain/sketches/cck49wDub ๐น๏ธ Real Time Waveform: https://editor.p5js.org/codingtrain/sketches/aaRIT-x6a
๐ฅ Previous: https://youtu.be/g3-PXyF8U70?list=PLRqwX-V7Uu6ZiZxtDDRCi6uhfTH4FilpH ๐ฅ All: https://www.youtube.com/playlist?list=PLRqwX-V7Uu6ZiZxtDDRCi6uhfTH4FilpH
References: ๐ p5.2 Reference: https://beta.p5js.org ๐ Introducing Whisper: https://cdn.openai.com/papers/whisper.pdf ๐ Model Cards for Model Reporting: https://arxiv.org/abs/1810.03993 ๐ Open Neural Network Exchange: https://onnx.ai ๐ Onnx-community Whisper-tiny.en model: https://huggingface.co/onnx-community/whisper-tiny.en ๐ Xenova: https://github.com/xenova ๐ Transformers.js: https://huggingface.co/docs/transformers.js/installation ๐ Announcing the new p5.sound.js library!: https://medium.com/processing-foundation/announcing-the-new-p5-sound-js-library-42efc154bed0 ๐ getUserMedia() documentation: https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia ๐ MediaRecorder() documentation: https://developer.mozilla.org/en-US/docs/Web/API/MediaRecorder ๐ Kokoro Repo: https://github.com/hexgrad/kokoro ๐ KokoroTTS Model: https://huggingface.co/hexgrad/Kokoro-82M ๐ ELIZA: https://en.wikipedia.org/wiki/ELIZA ๐ Rivescript: https://www.rivescript.com ๐ SmolLM3: https://huggingface.co/HuggingFaceTB/SmolLM3-3B ๐ Running models on WebGPU: https://huggingface.co/docs/transformers.js/guides/webgpu ๐ Using quantized models (dtypes): https://huggingface.co/docs/transformers.js/v3.8.1/guides/dtypes
Videos: ๐ https://youtu.be/0Ad5Frf8NBM ๐ https://youtu.be/KR61bXsPlLU
Live Stream Archives: ๐ด https://www.youtube.com/watch?v=KRDJAHArqaw
Related Coding Challenges: ๐ https://youtu.be/eGFJ8vugIWA ๐ https://youtu.be/8Z9FRiW2Jlc ๐ https://youtu.be/iFTgphKCP9U
Timestamps: 0:00:00 Hello! 0:00:35 Mapping out the pieces: speech-to-text, text-to-speech, and the brain 0:01:07 Thoughts on AI and creative exploration 0:02:44 Choosing the tools: Whisper and Kokoro TTS 0:04:06 Building a push-to-talk UI in p5.js 0:04:51 Finding models on Hugging Face with Transformers.js 0:05:36 About the Whisper model and model cards 0:06:55 Loading the Whisper pipeline in p5.js 0:09:04 Accessing the microphone with getUserMedia 0:10:44 Capturing audio with MediaRecorder 0:12:05 Processing audio chunks into a waveform 0:15:55 Speech-to-text working! 0:16:36 Building the chatbot brain (ELIZA-style therapist) 0:18:50 Setting up Kokoro TTS for text-to-speech 0:21:07 Playing synthesized audio with AudioBufferSource 0:23:41 Text-to-speech working! 0:25:32 Handling playback events 0:26:56 Swapping in a RiveScript number-guessing brain 0:31:22 Adding a language model (SmolLM2) as the brain 0:38:33 Final demo: the random number chatbot 0:39:03 Goodbye!
Editing by Mathieu Blanchette Animations by Jason Heglund Music from Epidemic Sound
๐ Website: https://thecodingtrain.com/ ๐พ Share Your Creation! https://thecodingtrain.com/guides/passenger-showcase-guide ๐ฉ Suggest Topics:โฆ