Extended Scan Response with BLE

Toit

#Extended Scan Response with BLE

Thread channel in help

Tej 11/16/2024 04:27 PM

AdvertisementData

is limited to 31 bytes in ble.toit. How does one get extended scan responses? I am unable to see the name or manufacturer-data of BLE beacons I am scanning. Here's a short code snippet I am using and the result.

import ble
import encoding.hex


BEACON-SERVICE ::= ble.BleUuid "FEAA"
SCAN-DURATION   ::= Duration --s=3

main:
  adapter := ble.Adapter
  central := adapter.central

  while true:
    addresses := Set
    data := []

    central.scan --duration=SCAN-DURATION: | device/ble.RemoteScannedDevice |
      if device.data.service-classes.contains BEACON-SERVICE:
        address := hex.encode device.address
        addresses.add (address.to-ascii-upper.trim --left "00")
        data.add device.data.manufacturer-data

    print addresses
    print data
    sleep Duration/5

Result:

{BC5729059011}
[#[]]

Env:

[tej@a1278 temperature_beacon]$ jag firmware
Device 'eloquent-fortune' is running Toit SDK v2.0.0-alpha.164

Running on ESP32-Wroom-32 (Olimex ESP32 Gateway)

What am I missing? I have looked through ble.toit and remote.toit and can't see any alternative scan methods nor any flags I can set.

floitsch 11/16/2024 04:29 PM

I'm currently traveling but will take a look next week (Tuesday or later).
If you have esp-idf code (in C) that does the right thing that would help, as I could compare it to what we do at the moment. (But not a requirement).

Tej 11/16/2024 05:36 PM

I think I have found the issue. Nimble does passive scans by default. I also see this in [resources/ble_esp32.cc](https://github.com/toitlang/toit/blob/6a2f27006dd25601046c7c996bbc3e159b10842c/src/resources/ble_esp32.cc#L2444):

/**
   * Perform a passive scan.  I.e., don't send follow-up scan requests to
   * each advertiser.
   */
  disc_params.passive = 1;

I have been using Bluedroid - let me compile a working example in Nimble and share it tomorrow.

toit/src/resources/ble_esp32.cc at 6a2f27006dd25601046c7c996bbc3e15...

Program your microcontrollers in a fast and robust high-level language. - toitlang/toit

floitsch 11/16/2024 05:38 PM

Great feedback. Thanks

Tej 11/17/2024 08:23 AM

Here's a minimal working example using Nimble with ESP-IDF. As I suspected we are not getting the scan response unless we set disc_params.passive = 0

https://github.com/tejpochiraju/nimble_beacon_scanner/blob/main/main/main.c

nimble_beacon_scanner/main/main.c at main · tejpochiraju/nimble_bea...

Contribute to tejpochiraju/nimble_beacon_scanner development by creating an account on GitHub.

floitsch 11/17/2024 09:12 AM

Thanks!
I will probably add a flag to the scan method, or, if I don't find any reason against it, enable active by default.

floitsch

Thanks! I will probably add a flag to the scan method, or, if I don't find any reason against it, enable active by default.

Tej 11/17/2024 09:20 AM

I would recommend keeping passive as the default and exposing a flag instead - consumes less energy in passive. And keeps the defaults consistent with upstream Nimble configuration.

I was trying to do just that locally and build a custom envelope but am currently running into network issues while running git submodule init - will try again later.

floitsch 11/17/2024 09:24 AM

When building locally (or even on GitHub builders) set the TOIT_GIT_VERSION to the version Jaguar/Artemis use. Otherwise the tooling will complain that the versions don't match. (Normally that's not completely safe, but I don't remember any recent breaking changes that would make the bytecodes incompatible)

Tej 11/18/2024 06:39 PM

I am not getting name or advertisement data on the Toit side of things even after changing disc_params.passive = 0; in src/resources/ble_esp32.cc. I verified that the modified ble_esp32.cc is being compiled by adding a few ESP_LOGE statements. Specifically, I changed the BleCentralManagerResource::_on_discovery function to print the raw event data (see attached file and output below). To be clear, disc_params.passive = 0; is working but the extra data is not being used.

I can see that the data is received in this function but is not being passed further? Also, not sure how to handle the raw advertisement data on the Toit side of things - this is not

manufacturer-data

but raw bytes instead.

E (7363) BLE: Discovery Event:
E (7363) BLE:   Address: 11:90:05:29:57:bc
E (7363) BLE:   RSSI: -61
E (7363) BLE:   Event Type: 0
E (7363) BLE:   Data Length: 29
E (7363) BLE:   Advertisement Data:
  02 01 06 03 03 aa fe 15 16 aa fe 21 04 0f 0b d4
  18 f8 43 0c 00 d6 ff f0 03 c4 01 1a 2e
E (7373) BLE: Discovery Event:
E (7383) BLE:   Address: 11:90:05:29:57:bc
E (7383) BLE:   RSSI: -62
E (7383) BLE:   Event Type: 4
E (7393) BLE:   Data Length: 24
E (7393) BLE:   Advertisement Data:
  09 16 80 20 66 09 00 00 00 00 0d 09 42 4c 45 30
  30 30 30 30 30 30 31 31

Steps:

TOIT_GIT_VERSION=v2.0.0-alpha.164 make esp32
# beacon.toit is the example code from the first post
toit compile --snapshot -o beacon.snapshot beacon.toit
toit tool firmware -e build/esp32/firmware.envelope container install beacon beacon.snapshot
toit tool firmware -e build/esp32/firmware.envelope flash --port /dev/ttyUSB0 --baud 921600
jag monitor

(edited)

floitsch 11/18/2024 07:17 PM

I will try to repro tomorrow. Thanks

floitsch 11/19/2024 09:44 AM

Started to look into this. I have a meeting in a few minutes, but here is my current status:
- I was easily able to run your modified code. Thanks!
- the Event Type for active scans seems to be 4.
- the received data (that has the debug prints) is stored in a DiscoveredPeripheral object and then put into a queue.
- the scan_next function (2480) then takes the next entry and passes it on to the Toit code.
- there is a ble_hs_adv_parse_fields (ble_esp32.cc:2502) that decomposes the received data into what is usually expected from advertisement packets.

If I understand you correctly, we shouldn't call ble_hs_adv_parse_fields for active-scan responses but pass the data directly to the Toit code. If that's the case I just need to store the event-type in the DiscoveredPerihperal object and then simply avoid the parsing-call.

Tej 11/19/2024 09:56 AM

Let me dig a bit more into ble_hs_adv_parse_fields before I comment on the behaviour. I am afraid I haven't had to dig into the details previously so I am also figuring out the "correct" behaviour.

floitsch 11/19/2024 09:58 AM

Looking too.

floitsch 11/19/2024 11:55 AM

After some reading, I'm pretty sure that the scan response packet follows the same structure as the advertisement packet.
In your example, there is a 09 16 80 20 66 09 00 00 00 00 which seems to be a service data - 16-bit UUID. It seems like the 16-bit UUID is 0x2080 and the data is 66 09 00 00 00 00.
After that, the 0d 09 42 4c 45 30 30 30 30 30 30 30 31 31 decodes as name BLE000000011.
It looks like we don't copy these fields to the Toit side yet.

If the macos-side allows it I will probably just move the decoding of the frame into Toit code. That would give access to the raw data, and also make it easier to implement the missing fields.

floitsch 11/19/2024 01:22 PM

@Tej

you changed the interval and the window.
Do you have some information on whether that's something that should be by default?
Should we expose it to the Toit level?

floitsch

@Tej you changed the interval and the window. Do you have some information on whether that's something that should be by default? Should we expose it to the Toit ...

Tej 11/19/2024 05:56 PM

I did a literal translation of the parameters from our current code base to Nimble. I just checked with interval and window = 0 (default) and there's no difference in the result. I doubt we need those and I suspect these are left over from some old experimentation.

In terms of exposing it, maybe make these optional with defaults as 0? Where does Toit sit on simplicity vs power?

OPTej

I did a literal translation of the parameters from our current code base to Nimble. I just checked with interval and window = 0 (default) and there's no difference in the result. I...

floitsch 11/19/2024 05:58 PM

Exposing it with defaults sounds good.

floitsch 11/19/2024 05:58 PM

We are not opposed to expose complicated things as long as it starts simple.

floitsch

After some reading, I'm pretty sure that the scan response packet follows the same structure as the advertisement packet. In your example, there is a `09 16 80 20 66 09 00 00 00 00...

Tej 11/19/2024 05:58 PM

I am not clear on what you mean by moving the decoding of the frame into Toit code? Would a developer have access to the raw bytes? Reason I ask is, there's an increasing number of BLE sensors and I am not sure all of them allow standard decoding.

OPTej

I am not clear on what you mean by moving the decoding of the frame into Toit code? Would a developer have access to the raw bytes? Reason I ask is, there's an increasing number of...

floitsch 11/19/2024 06:00 PM

Yes. I have started to refactor the code so that the packet data (31 bytes) is given to the Toit code (the BLE library). I can now implement the decoding there.

floitsch 11/19/2024 06:00 PM

And I would expose the raw data. Most users wouldn't need it, but it shouldn't hurt.

floitsch 11/19/2024 06:01 PM

That said. I'm actually splitting it up into fields right now. So basically a partial decoding.

Tej 11/19/2024 06:02 PM

Not sure this is directly useful here but here's an Aruba document that details how they handle BLE advertisement.

https://www.arubanetworks.com/techdocs/Instant_87_WebHelp/Content/instant-ug/custom-iap-param/ble-iot.htm

I get why they added support for "known" beacons (business partnerships) but that's a losing battle as the number of beacons increases daily. We find their raw frames more useful.

OPTej

Not sure this is directly useful here but here's an Aruba document that details how they handle BLE advertisement. https://www.arubanetworks.com/techdocs/Instant_87_WebHelp/Conte...

floitsch 11/19/2024 06:03 PM

The content of a ble advertising packed it well defined. So we could decode it in C and send the decoded data to Toit. I find it just more cumbersome

floitsch

That said. I'm actually splitting it up into fields right now. So basically a partial decoding.

Tej 11/19/2024 06:03 PM

This is the bit I am unclear on the boundary between standard and common practice. Were you able to find a standard on the fields?

OPTej

This is the bit I am unclear on the boundary between standard and common practice. Were you able to find a standard on the fields?

floitsch 11/19/2024 06:03 PM

Yes

floitsch 11/19/2024 06:04 PM

In the Bluetooth spec

Tej 11/19/2024 06:08 PM

Cool. I had assumed the payloads from a couple of our vendors were not standards compliant but reading up, looks like they are.

Tej 11/20/2024 03:23 PM

I have been a bit tied up so haven't had a chance to dig further. Hoping for some time tomorrow.

floitsch 11/20/2024 03:25 PM

No need. I'm busy refactoring that code. I'm confident it will solve the case you showed earlier.

Tej 11/23/2024 05:28 AM

Hi @floitsch - I have time this weekend so happy to support with this as you need.

floitsch 11/23/2024 06:05 AM

Thanks for the offer. I'm making good progress and I think I will have some PRs ready on Monday. Just writing the tests took much longer than I anticipated.

Tej 11/23/2024 07:10 AM

No worries, I have handed off the primary code migration tasks to one of our engineers so my hands are bit freer to tackle general/common needs like BLE, WiFi (AP-STA switchover) and OTA. By the end of the year, I expect we would have moved 3 if not 4 of our products to Toit. That's 3 years of C compressed into 1.5 months of Toit!

OPTej

Hi @floitsch - I have time this weekend so happy to support with this as you need.

floitsch 11/24/2024 06:31 PM

FYI: PRs have been uploaded:
https://github.com/toitlang/toit/pull/2641
https://github.com/toitlang/toit/pull/2642
https://github.com/toitlang/toit/pull/2643
https://github.com/toitlang/toit/pull/2644

Improve BLE advertisements. by floitsch · Pull Request #2641 · toit...

Split out the fields and make them accessible.
This makes it possible to move more code into Toit, and also gives more control to users that need more advanced advertisement fields.

Summary by Cod...

Support active BLE scans. by floitsch · Pull Request #2642 · toitla...

Active scans send a request to the device when they have received an advertisement.

Give BLE scan more options. by floitsch · Pull Request #2643 · toit...

Most users shouldn't touch them, but for power users it's nice to have them available.

Cleanups in the BLE library. by floitsch · Pull Request #2644 · toi...

Better names...

floitsch 11/24/2024 06:31 PM

Let me know if they solve your issues. (Once they have been committed).

Tej 11/25/2024 01:05 AM

Thanks! Will give it a try.

bitphlipphar 11/26/2024 01:36 PM

Jaguar v1.45.0 comes with the new SDK that bundles all the changes.

bitphlipphar 11/29/2024 06:07 AM

@Tej

Did you get a chance to try the changes?

Tej 11/30/2024 02:13 PM

@bitphlipphar

will test this tomorrow. Got tied up in something.

Tej 12/01/2024 04:59 PM

Have set aside time to do more thorough testing tomorrow as I couldn't quite get the scan working correctly with:

import ble
import encoding.hex


BEACON-SERVICE ::= ble.BleUuid "FEAA"
SCAN-DURATION   ::= Duration --s=-1

main:
  adapter := ble.Adapter
  central := adapter.central

  while true:
    addresses := Set
    data := []

    central.scan --duration=SCAN-DURATION 
                 --active=true 
                 --limited-only=false
                 --interval=160 
                 --window=158: 
                 | device/ble.RemoteScannedDevice |
      if device.data.services.contains BEACON-SERVICE:
        address := hex.encode device.address-bytes.stringify
        print device.identifier
        print device.address-bytes
        print device.address-type
        print device.data
    sleep Duration/5

Output:

an instance with class-id 55
#[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
#[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
0

## Notes

1. --limited-only=true does not give a result at all - whereas this parameter did not seem to have any effect in my Nimble example.
2. Changing interval and window did not yield data either.

Tomorrow I will build from source and print logs from the C layer to dig deeper.

floitsch 12/01/2024 05:00 PM

Interesting. Curious about your findings.
Thanks for testing.

Tej 12/02/2024 04:56 PM

I think I misinterpreted the data structure. I am infact getting the data correctly but not the name.

// From C
E (6538) BLE: Discovery Event:
E (6538) BLE:   Address: 11:90:05:29:57:bc
E (6538) BLE:   RSSI: -89
E (6538) BLE:   Event Type: 0
E (6538) BLE:   Data Length: 29
E (6538) BLE:   Advertisement Data:
  02 01 06 03 03 aa fe 15 16 aa fe 21 04 0f 0b d4
  17 29 45 b6 ff e1 ff d2 03 f2 01 29 db
E (6548) BLE: Discovery Event:
E (6558) BLE:   Address: 11:90:05:29:57:bc
E (6558) BLE:   RSSI: -86
E (6558) BLE:   Event Type: 4
E (6568) BLE:   Data Length: 24
E (6568) BLE:   Advertisement Data:
  09 16 80 20 66 09 00 00 00 00 0d 09 42 4c 45 30
  30 30 30 30 30 30 31 31

// From Toit
ID: #[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address bytes: #[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address type: 0
Name: null
Data: #[0x02, 0x01, 0x06, 0x03, 0x03, 0xaa, 0xfe, 0x15, 0x16, 0xaa, 0xfe, 0x21, 0x04, 0x0f, 0x0b, 0xd4, 0x17, 0x29, 0x45, 0xb6, 0xff, 0xe1, 0xff, 0xd2, 0x03, 0xf2, 0x01, 0x29, 0xdb]

The ESP_LOGE statements are the same as my ble_esp32.cc attachment from earlier. The Toit code is:

print "ID: " + device.identifier.stringify
print "Address bytes: " + device.address-bytes.stringify
print "Address type: " + device.address-type.stringify
print "Name: " + device.data.name.stringify
print "Data: " + device.data.to-raw.stringify

Given the name comes from the second scan event (type 4), is it possible that the response from this event is not being copied over?(edited)

floitsch 12/02/2024 04:57 PM

I will try to have a look later today or early tomorrow.
If there isn't any update tomorrow please remind me.

floitsch 12/03/2024 09:14 AM

Starting to look at this.
In theory you should get 2 different RemoteScannedDevice objects in Toit. It would be up to you to merge them.(edited)

floitsch 12/03/2024 09:16 AM

You can use is-scan-response to distinguish between the two: https://libs.toit.io/ble/remote/class-RemoteScannedDevice#is-scan-response

OPTej

I think I misinterpreted the data structure. I am infact getting the data correctly but not the name. ``` // From C E (6538) BLE: Discovery Event: E (6538) BLE: Address: 11:90:0...(edited)

bitphlipphar 12/03/2024 10:03 AM

You may want to get used to print "ID: $device.identifier" instead of the manual stringification and concatenation with +

floitsch

Starting to look at this. In theory you should get 2 different RemoteScannedDevice objects in Toit. It would be up to you to merge them.(edited)

Tej 12/03/2024 05:20 PM

Got it. I think I missed that in the docs. Unclear to me how I would go about the merge, though. This is probably mostly because I am yet to internalise the Toit way of working (see above!).

Will I get multiple RemoteScannedDevice entries. i.e. a List? Or will the block get called on each response, i.e. once for the Discovery event and once for the Scan Response?

If it's the latter, I should have seen the data in my prints at least once but I didn't. I will refactor the code to do the merge and try again.

floitsch 12/03/2024 05:20 PM

In theory your block should have been called twice; once for each response.

floitsch 12/03/2024 05:24 PM

It's not documented, but you can easily merge two advertisements by just merging their data-block lists.
From memory this should be something like:

scanned-device1 := ...
scanned-device2 := ...
merged-blocks := scanned-device1.data.data-blocks + scanned-device2.data.data-blocks
merged-advertisement := Advertisement merged-blocks --no-check-size
print merged-advertisement.name
...

(edited)

floitsch 12/03/2024 05:25 PM

And I will update the doc (both for the scan-result as well as how to merge them).

Tej 12/03/2024 06:04 PM

Definitely getting multiple callbacks (count depends on duration) but only for Discovery.

ID: #[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address bytes: #[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address type: 0
Name: null
Data: #[0x02, 0x01, 0x06, 0x03, 0x03, 0xaa, 0xfe, 0x11, 0x16, 0xaa, 0xfe, 0x20, 0x00, 0x0b, 0xd2, 0x17, 0x33, 0x00, 0x4b, 0xe8, 0x01, 0x
01, 0xf8, 0x3e, 0x16]
Is Scan Response: false

ID: #[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address bytes: #[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address type: 0
Name: null
Data: #[0x02, 0x01, 0x06, 0x03, 0x03, 0xaa, 0xfe, 0x15, 0x16, 0xaa, 0xfe, 0x21, 0x04, 0x0f, 0x0b, 0xd2, 0x17, 0x33, 0x47, 0xf3, 0xff, 0x
e1, 0xff, 0xd2, 0x03, 0xf2, 0x01, 0x2b, 0x06]
Is Scan Response: false

// Repeats...

I simplified the code to do a single scan:

import ble
import encoding.hex


BEACON-SERVICE ::= ble.BleUuid "FEAA"
SCAN-DURATION   ::= Duration --s=1

main:
  adapter := ble.Adapter
  central := adapter.central

  central.scan --duration=SCAN-DURATION 
                --active=true 
                --limited-only=true
                --interval=160 
                --window=158: 
                | device/ble.RemoteScannedDevice |
    if device.data.services.contains BEACON-SERVICE:
      address := hex.encode device.address-bytes.stringify
      print "ID: $device.identifier"
      print "Address bytes: $device.address-bytes"
      print "Address type: $device.address-type"
      print "Name: $device.data.name"
      print "Data: $device.data.to-raw"
      print "Is Scan Response: $device.is-scan-response"

Can you point me to where you pass the data to Toit from ble_esp32.cc - I can try and debug further from there.

floitsch 12/03/2024 06:12 PM

Interesting.
The discovered peripheral is created here: https://github.com/toitlang/toit/blob/9bc1f09a2da28139d598162f17182479a9351e0c/src/resources/ble_esp32.cc#L1725
We then take them out of the queue in the scan_next function here:
https://github.com/toitlang/toit/blob/9bc1f09a2da28139d598162f17182479a9351e0c/src/resources/ble_esp32.cc#L2485

I think the reason it doesn't work for you is that the scan-response doesn't have the service (BEACON-SERVICE) anymore. If you look at the raw data, you can see that the first advertisement data contains the service (aa fe), but the scan response doesn't.
So you would need to collect the identifier (discovered-device.identifier) and then allow also scanned-device objects that don't have the BEACON-SERVICE but where you know that they have that service.
Something like:

found-devices := {}
central.scan ...
  if device.data.contains-service BEACON-SERVICE or found-devices.contains device.identifier:
    found-devices.add device.identifier
    ...

toit/src/resources/ble_esp32.cc at 9bc1f09a2da28139d598162f17182479...

Program your microcontrollers in a fast and robust high-level language. - toitlang/toit

Tej 12/05/2024 04:06 AM

Well, that was daft. You were right about BEACON-SERVICE being missing from the scan response.

ID: #[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address bytes: #[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address type: 0
Name: null
Data: #[0x02, 0x01, 0x06, 0x03, 0x03, 0xaa, 0xfe, 0x11, 0x16, 0xaa, 0xfe, 0x20, 0x00, 0x0b, 0xd2, 0x18, 0x02, 0x00, 0x4f, 0xa9, 0x65, 0x02, 0x0b, 0x0f, 0xd7]
Is Scan Response: false

ID: #[0x00, 0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address bytes: #[0xbc, 0x57, 0x29, 0x05, 0x90, 0x11]
Address type: 0
Name: BLE000000011
Data: #[0x09, 0x16, 0x80, 0x20, 0x66, 0x09, 0x00, 0x00, 0x00, 0x00, 0x0d, 0x09, 0x42, 0x4c, 0x45, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x30, 0x31, 0x31]
Is Scan Response: true

Happy to report this is working as needed. Thanks again for your patience and support!

53 messages in total