A Voice Bot Mobile App

Dialogflow In Flutter

Original (and outdated) instructions for this app. My implementation in this article.

Step #1: Create Dialogflow ES agent

1. Make sure it uses Auto Speech Adaptation when creating the agent.
2. Enable “Dialogflow API” in the GCP console. Create a service account under credentials and grant Dialogflow API Admin Role.
3. Add JSON new key to the service account just created and store it locally on your computer.

Step #2: Create Audio App

1. Install Flutter and Android Studio. Run “flutter create voicebot”.
2. Set SDK versions for the sound_stream package in android/app/build.gradle: minimum 21 and target 30.
3. Give permission to the Internet and the microphone in app/src/main/AndroidManifest.xml.
4. Install sound_stream, rxdart, dialogflow_grpc
5. Move the JSON new key file from the last step into an assets folder. Be careful with the key.
6. Connect your physical phone using scrcpy and run the app

Step #3: Create a chat UI with STT support

On a high level, the widget tree is MyHomePage => chat(stateful) => message(stateless). STT is Speech-To-Text.

Chat Widget

It lays out the chat UI and handles Dialogflow. The UI has

1. ListView to display all message bubbles.
2. TextField for typing.
3. IconButton to send a message. When it’s pressed, we need to submit the text.
4. IconButton to use mic. When it’s pressed, we need to stop or handle the audio stream.

We need a few states:

  final List<ChatMessage> _messages = <ChatMessage>[];
  final TextEditingController _textController = TextEditingController();
  late DialogflowGrpcV2Beta1 dialogflow;
  bool _isRecording = false; // if mic is recording
  RecorderStream _recorder = RecorderStream();
  StreamSubscription? _recorderStatus;
  StreamSubscription<List<int>>? _audioStreamSubscription;
  BehaviorSubject<List<int>>? _audioStream;

1. get Dialogflow client through the service account in initState(). This is done using the credential json we downloaded Dialogflow API.
2. To stopStream, we stop RecorderStream, cancel StreamSubscription, and close _audioStream(BehaviorSubject)
3. To handleStream, you can decide the dialog phrases (application-specific), audio config (sample hertz, encoding), then get intent by listening to dialogflow.streamingDetectIntent() given the audio stream.
4. To submitText, you get response from dialogflow.detectIntent() given the text and update the message bubble list in UI.

Message Widget

Each message has two horizontal parts: CircleAvatar and the message text. Optionally, the message text can be a column to stack the sender’s initials on top of the text. The only difference between bot’s message and your message is bot’s has CrossAxisAlignment.start to align on the left while yours has it as CrossAxisAlignment.end to align on the right

Step #4: Create intents

Add intents and responses. You can also use Knowledge Connector.

If you like this tutorial, you may also find 3 Levels of Flutter Tests helpful. They are part of my Become Flutter Comfortable in 23 Days series

Reference

1. Dialogflow product
2. Dialogflow Intent
3. Dialogflow Knowledge connectors