MQTT Discovery in Home Assistant

In previous notes we have explored how to integrate an IoT device (ESP8266) with an MQTT broker. The next step on this journey is to actually make such a device controllable from Home Assistant via the MQTT connection.

Rough Plan Of Action

Only the first option would be relatively “automatic” or “plug and play”, in the sense that one could point the device at the MQTT broker and (presumably) have it automatically “appear” in Home Assistant. The latter two options it appears would require either manually setting the configuration YAML in Home Assistant or manually entering some stuff in the UI. Additionally the “discovery” approach is taken by the Tasmota integration which is what I’m used to and I find works pretty well.

There appear to be two options to send the discovery messages at the right time, which are following “birth” and “will” messages from Home Assistant, or publishing those discovery messages with the “retained” flag. Because we have explored subscribing to topics already in the previous notes and because there seem to be more disadvantages to the latter approach, in this note we’ll take the former approach.

Step 3a - we’ll need to send a discovery message when the device connects to the MQTT broker

Sending Discovery Messages On The Right Topic

<discovery_prefix> is a configuration item in the MQTT integration in Home Assistant, which defaults to homeassistant. I have not changed this in my Home Assistant, hence that’s the right value to use.

For <component> we’ll use device since we’ll be using the “newer” device discovery rather than component discovery.

Per the documentation, since we will assign our devices a unique_id, we can omit the <node_id> level.

Per the documentation, since we will assign our devices a unique_id, the <object_id> will be set to that unique_id.

So for the case where the unique_id is e.g. e1e080f341364ceeb6038a050ed7dd60 the discovery topic would be homeassistant/device/e1e080f341364ceeb6038a050ed7dd60/config.

Find The Right Format For Discovery Messages

This is the major unknown in this note, everything else is frankly pretty easy to figure out with the code given in previous notes.

Unfortunately, the documentation is not (at least in my opinion) particularly well organised, so here’s what I’m able to piece together.

command_topic, string, should be given for a device which accepts commands, such as a switch or a light.

state_topic, string, should be given for a device which can publish states to Home Assistant such as a switch (on/off) or a sensor.

qos, integer, optional, I assume it sets the quality of service for messages sent to the device? This is apparently not documented on the page.

encoding, string, optional, appears to tell Home Assistant the string encoding expected in messages from and to the device, which defaults to UTF-8. UTF-8 “covers” the entire range of ASCII encoding, outside of which our device will not step, so we can ignore this key completed.

components is a map, with a key per “component” in the device, each with an map value. A component is an independent “part” in a device - for example a single switch might just have one switch component, a temperature and humidity sensor might have two components i.e. two sensors with different units of measurement,

* where abbreviated keys are supported, I’ll give full_key / abbreviated_key

** JSON objects, but I’ll call them maps because that comes more naturally to me

device

To find the full documentation for the device map I had to dive into the YAML configuration documentation for the specific MQTT integrations, for example buttons or switches.

I’m pretty sure the device entry is the same for all of the different integrations, but I don’t feel like comparing all 20 or however many there are…

Everything in this map is apparently optional, which is a bit surprising considering the device key is required in the device discovery payload?

So, I’m just going to start with whatever is suggested in the example discovery payloads, under the assumption that nothing here (except probably the ids) is that important for our non-commercial device anyway. These fields are

For identifiers we’ll just use the unique id for the device, which we’ll take as a UUID. We’ll use that as serial_number too.

For name we worked in previous notes on a switch, so let’s call it “Test Switch”.

For sw_version and hw_version I’m sure a real project would follow some well defined versioning scheme such as semver. For simple learning projects like this, I prefer a simpler approach, so we’ll just use an integer number, setting these both to (string) “1”.

Assuming UUID c1d9bca9ddfc4803be31d7920d57f91e then would give us a device map

{
    "hw_version": "1",
    "identifiers": "c1d9bca9ddfc4803be31d7920d57f91e",
    "name": "Test Switch",
    "serial_number": "c1d9bca9ddfc4803be31d7920d57f91e",
    "sw_version": "1"
 }

origin

The purpose of this field it seems is only to add context to the logs generated by Home Assistant relating to updates to the device, but it’s required for the device discovery payload, so add it we must. There are three keys

Only name is required so that’s all we’ll give, and we’ll make it match the device name, Test Switch.

{
    "name": "Test Switch"
}

command_topic and state_topic

These fields tell Home Assistant where it should publish commands to be consumed by the device and on which topic the device will publish updates about its state.

Luckily we have chosen a switch example to work on, which means we will need to set both.

Obviously these need to be unique so multiple devices don’t interfere with one another.

However I would generally go with something a bit more structured and that provides a bit more context - remember in the previous note we saw the Tasmota topics like stat/tasmota_DEVICE_SHORT_ID/RESULT and cmnd/tasmota_DEVICE_SHORT_ID/Power1.

We might want to include the application name or the manufacturer name (if we had one) or the model name, but since we only have a device name for now, let’s use that to “namespace” our topics.

components

After reading through twice, I can’t see where this is explicitly spelled out, but I think each value in the map under components must be a configuration from the lists in this section.

{
  "p": "sensor",
  "device_class":"temperature",
  "unit_of_measurement":"°C",
  "value_template":"{{ value_json.temperature}}",
  "unique_id":"temp01ae_t"
}

There is a quite an array of configuration entries for a switch, the full example from the documentation is

mqtt:
  - switch:
      unique_id: bedroom_switch
      name: "Bedroom Switch"
      state_topic: "home/bedroom/switch1"
      command_topic: "home/bedroom/switch1/set"
      availability:
        - topic: "home/bedroom/switch1/available"
      payload_on: "ON"
      payload_off: "OFF"
      state_on: "ON"
      state_off: "OFF"
      optimistic: false
      qos: 0
      retain: true

Luckily, almost all of this is optional, and some of it (e.g. the command_topic and switch_topic) will be provided at the device level when using device discovery.

Actually there is one missing configuration item from that YAML, which is only supported for discovery messages, and which is documented for the device discovery payload. That is platform / p. The platform is the component type, the same as the key against which you’ll find the configuration map in the YAML, in this case switch. This must be provided for discovery, so we can start with this JSON.

{
    "platform": "switch"
}

unique_id is also documented as being required when using device discovery. We’ll use a new UUID for this.

{
    "platform": "switch",
    "unique_id": "<<SOME NEW UUID>>"
}

Everything else is optional and I think we can do without it in the first instance - I won’t give a justification for each, because there are quite a lot, except to say that the defaults are given where required & reasonable and for name which is already defined at the device level.

{
    "platform": "switch",
    "unique_id": "a2f9f04b8ea04993b7e0d71cb53162d6"
}

Full Device Discovery Payload

{
    "command_topic": "test_switch/command/c1d9bca9ddfc4803be31d7920d57f91e",
    "components": {
        "a2f9f04b8ea04993b7e0d71cb53162d6": {
            "platform": "switch",
            "unique_id": "a2f9f04b8ea04993b7e0d71cb53162d6"
        }
    },
    "device": {
        "hw_version": "1",
        "identifiers": "c1d9bca9ddfc4803be31d7920d57f91e",
        "name": "Test Switch",
        "serial_number": "c1d9bca9ddfc4803be31d7920d57f91e",
        "sw_version": "1"
    },
    "origin": {
        "name": "Test Switch"
    },
    "state_topic": "test_switch/state/c1d9bca9ddfc4803be31d7920d57f91e"
}

Checking With Home Assistant

Using the mqttx CLI we can directly send this payload to our MQTT broker attached to Home Assistant to see if Home Assistant accepts it.

Recall that MQTT_USER and MQTT_PASSWORD are the username and password set up in Home Assistant for connections to the MQTT broker, in my case I’ll be using a user I created for tasmota devices.

Amazingly, this worked on the first try! The switch appeared under the MQTT Integrations entry.

Finally, as we have done in the previous Tasmota related note, we can send state updates on the state topic and we will see the state in Home Assistant change from unknown to on/off.

Then, to double check we’re not creating a mess forever, I restarted Home Assistant to check that the test switch device disappears, which I believe it should since the discovery message was not published as retained. Alas, it did not! Luckily it can be manually deleted from the MQTT integration.

I checked that resending the discovery message makes the device reappear under the MQTT integration too.

Send A Discovery Message When The Device Connects

Based on our previous MQTT notes, this should be quite straightforward. We’ll start with a template based on the code we developed before. I’ve added comments in to refresh our memories about what the various parts do, and I’ve inserted TODOs where we’ll be implementing new or additional functionality.

#include <ESP8266WiFi.h>
#include <PubSubClient.h>

// Enter your WiFi credentials here
const char* connect_to_ssid = "your-wifi-ssid-here";
const char* connect_to_password = "your-wifi-password-here";

// Enter your mqtt connection details & mqtt user credentials here
const char* mqtt_host = "your-mqtt-host-here";
const uint16_t mqtt_port = 1833;
const char* mqtt_user = "your-mqtt-user-here";
const char* mqtt_password = "your-mqtt-password-here";
// Note this has been updated to match the device's id
const char* mqtt_client_id = "c1d9bca9ddfc4803be31d7920d57f91e";

WiFiClient espClient;
PubSubClient client(espClient);

// Buffer for outgoing messages, and its size including null terminator
const uint16_t maximum_outgoing_message_length = 20;
char outgoing_message[maximum_outgoing_message_length];

// Buffer for incoming messages, and its size including null terminator
const uint16_t maximum_incoming_message_length = 20;
char incoming_message[maximum_incoming_message_length];

// When a message is received, its length including null terminator is stored here
volatile uint8_t stored_message_length = 0;

// Invoked when a message is received on a topic that we're subscribed to.
// Stores the message in incoming_message and the size of that message
// in stored_message_length. If the received message is larger than
// the maximum size of the incoming message buffer, the message is truncated to fit.
void callback(char* topic, byte* payload, unsigned int length) {
  unsigned int truncated_length = maximum_incoming_message_length - 1;
  if (length < truncated_length) {
    truncated_length = length;
  }
  for (uint8_t i=0; i<truncated_length; i++) {
    incoming_message[i] = (char)payload[i];
  }
  incoming_message[truncated_length] = '\0';
  stored_message_length = truncated_length;
  Serial.print("Topic ");
  Serial.print(topic);
  Serial.print(" received '");
  Serial.print(incoming_message);
  Serial.print("' length ");
  Serial.print(stored_message_length);
  Serial.println("");
}

// Retries connecting to the MQTT broker until
// the MQTT client indicates that it is connected
void ensure_mqtt_connected() {
  while (!client.connected()) {
    delay(250);
    Serial.print("Reconnecting to MQTT at ");
    Serial.print(mqtt_host);
    Serial.print(":");
    Serial.println(mqtt_port);
    client.setServer(mqtt_host, mqtt_port);
    bool connected = client.connect(mqtt_client_id, mqtt_user, mqtt_password);
    Serial.print("MQTT connected? ");
    Serial.print(connected);
    Serial.print(" Code ");
    Serial.println(client.state());
    client.setCallback(callback);
    // TODO: set up subscription to correct topic(s)
    bool subscribed = client.subscribe("TODO");
    Serial.print("MQTT subscribed? ");
    Serial.println(subscribed);
  }
}

// Set up the serial output and wifi connection
void setup() {
  Serial.begin(115200);
  Serial.println("");
  Serial.print("Connecting to ");
  Serial.println(connect_to_ssid);
  WiFi.mode(WIFI_STA);
  WiFi.begin(connect_to_ssid, connect_to_password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.print(".");
  }
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());

  ensure_mqtt_connected();

  // TODO: send discovery message
}

void loop() {
  ensure_mqtt_connected();

  // Receive messages on subscribed topics
  client.loop();

  // TODO: echo state changes 
}

The first TODO is to send the discovery message on connect. From our previous MQTT notes, sending a message looks like this.

  if (stored_message_length > 0) {
    for (uint8_t i=0; i<stored_message_length; i++) {
      outgoing_message[i] = incoming_message[i];
    }
    outgoing_message[stored_message_length] = '\0';
    Serial.print("About to send message: ");
    Serial.println(outgoing_message);
    bool published = client.publish("state/device/test", outgoing_message);
    Serial.print("Successfully published? ");
    Serial.println(published);
    stored_message_length = 0;
  }

We’re going to be sending different types of messages (discovery and state), plus we’re going to be doing that from various different points in the code, so we’re going to want first to move this into a reusable function.

// publish to mqtt on given topic a message containing given content.
// content must be null terminated.
void send_mqtt_message(const char* topic, const char* content) {
  Serial.print("About to send message: ");
  Serial.println(content);
  bool published = client.publish(topic, content);
  Serial.print("Successfully published? ");
  Serial.println(published);
}

We should then define the discovery message in the top matter. This is dense and hard to read, I’ve de-prettified (or compacted) the JSON with jq -c .

const char* discovery_message = "{\"command_topic\":\"test_switch/command/c1d9bca9ddfc4803be31d7920d57f91e\",\"components\":{\"a2f9f04b8ea04993b7e0d71cb53162d6\":{\"platform\":\"switch\",\"unique_id\":\"a2f9f04b8ea04993b7e0d71cb53162d6\"}},\"device\":{\"hw_version\":\"1\",\"identifiers\":\"c1d9bca9ddfc4803be31d7920d57f91e\",\"name\":\"Test Switch\",\"serial_number\":\"c1d9bca9ddfc4803be31d7920d57f91e\",\"sw_version\":\"1\"},\"origin\":{\"name\":\"Test Switch\"},\"state_topic\":\"test_switch/state/c1d9bca9ddfc4803be31d7920d57f91e\"}";

const char* discovery_topic = "homeassistant/device/c1d9bca9ddfc4803be31d7920d57f91e/config";

Now we can put these pieces together in the setup code to send the discovery message.

// Set up the serial output and wifi connection
void setup() {
  // ...

  send_mqtt_message(discovery_topic, discovery_message);
}

First ensuring that the Test Switch device is removed from the MQTT integration in Home Assistant, then uploading this code and running it on the ESP8266, we will see the expected serial output

and the Test Switch device will appear in the MQTT integration exactly as it did before when publishing with mqttx.

Send A Discovery Message When Home Assistant Sends Online Message

I find the expected behaviour of MQTT integration devices on restart (I think that’s the right documentation section) quite unclear. The documentation suggests that the devices added via the MQTT integration should be “unavailable” after a restart until the discovery message has been received again.

And then, after restarting the ESP8266 and letting it send the discovery message again, the device seems to become available again (but in unknown state, which we’ll fix later)

So, exactly as the linked documentation section describes, we will need a way to re-send the discovery message when Home Assistant is restarted. The most conventional way suggested to handle this situation is to listen for Home Assistant’s Online/Offline messages and re-send the discovery message on the former.

First of all, let’s see if we can capture offline/online messages with mqttx.

Then after all of the “Starting Bluetooth …”, “Starting MQTT …”, etc… messages have passed in the Home Assistant app, we see

Hence we’ll need to subscribe to the homeassistant/status topic and listen for the message which says simply online.

Looking forward, we know for this switch device that we will also at some point need to subscribe to the command channel so that Home Assistant can control the state of the switch. This means we will need to subscribe to multiple topics, which is something we have not done before in the previous notes. The pub sub client documentation page seems to suggest that we can call subscribe on the MQTT client multiple times to subscribe to multiple topics. So let’s move the subscription method calls into our own function together with our handy serial logging.

// subscribe the MQTT client to the given topic
void subscribe_mqtt_topic(const char* topic) {
    bool subscribed = client.subscribe(topic);
    Serial.print("MQTT subscribed? ");
    Serial.println(subscribed);
}

// Retries connecting to the MQTT broker until
// the MQTT client indicates that it is connected
void ensure_mqtt_connected() {
  while (!client.connected()) {
    // ...
    subscribe_mqtt_topic("homeassistant/status");
    // TODO: subscribe to command topic
  }
}

The callback we currently have set up just prints the message and the topic to serial, so we can first upload our updated code and restart Home Asssistant again to test we’re receiving those status messages. We’ll this printed in the serial monitor

MQTT Home Assistant Discovery

Rough Plan Of Action

Sending Discovery Messages On The Right Topic

Find The Right Format For Discovery Messages

`device`

`origin`

`command_topic` and `state_topic`

`components`

Full Device Discovery Payload

Checking With Home Assistant

Send A Discovery Message When The Device Connects

Send A Discovery Message When Home Assistant Sends Online Message

MQTT Home Assistant Discovery

Rough Plan Of Action

Sending Discovery Messages On The Right Topic

Find The Right Format For Discovery Messages

device

origin

command_topic and state_topic

components

Full Device Discovery Payload

Checking With Home Assistant

Send A Discovery Message When The Device Connects

Send A Discovery Message When Home Assistant Sends Online Message

`device`

`origin`

`command_topic` and `state_topic`

`components`