several Toit application can not send data concurrently to adafruit+mqtt

Toit

#several Toit application can not send data concurrently to adafruit+mqtt

Thread channel in help

Shadi 11/24/2022 01:35 PM

I have 3 different sensors and try to send their data over wifi to adafruit feeds. but when I try to run them, just one of them can send data and other services will not send data

floitsch 11/24/2022 02:47 PM

How do you use the MQTT client?

OPShadi

I have 3 different sensors and try to send their data over wifi to adafruit feeds. but when I try to run them, just one of them can send data and other services will not send data

floitsch 11/24/2022 02:47 PM

Are there error messages?

floitsch 11/24/2022 02:48 PM

You probably want to have just one client that is used by all sensors.

floitsch 11/24/2022 02:48 PM

Otherwise you need three connections which could be expensive (resource-wise).

Shadi 11/24/2022 03:00 PM

I did not recive any error, but when I check my feeds I found by starting the second service, the first one was stoped to send data.

floitsch 11/24/2022 03:00 PM

ah.

floitsch 11/24/2022 03:00 PM

Do you maybe use the same client id?

floitsch 11/24/2022 03:01 PM

If you have two clients with the same ID, then the broker will automatically disconnect the first one.

floitsch

Do you maybe use the same client id?

Shadi 11/24/2022 03:02 PM

oh, I do not have a client id in my appilications

floitsch

If you have two clients with the same ID, then the broker will automatically disconnect the first one.

Shadi 11/24/2022 03:03 PM

But, I think you are right, I have to find out the client id

floitsch 11/24/2022 03:03 PM

I think that means that you should get a random one. In theory that would allow multiple clients.

floitsch 11/24/2022 03:03 PM

Can you show me the code you use to create your client?

Shadi 11/24/2022 03:03 PM

yes

floitsch 11/24/2022 03:04 PM

don't forget to remove the IO_KEY (and maybe IO_USERNAME)

Shadi 11/24/2022 03:04 PM

transport := mqtt.TcpTransport.tls network --host=HOST
--root_certificates=[ certificate_roots.DIGICERT_GLOBAL_ROOT_CA ]
client := mqtt.Client --transport=transport
sensor := dht11.Dht11 (gpio.Pin PIN_NUMBER)
options := mqtt.SessionOptions
--client_id = "toit-example-client"
--username = ADAFRUIT_IO_USERNAME
--password = ADAFRUIT_IO_KEY

client.start --options=options

Shadi 11/24/2022 03:04 PM

for both services I use this code

floitsch 11/24/2022 03:04 PM

There it is.

floitsch 11/24/2022 03:04 PM

The "client_id"

Shadi 11/24/2022 03:05 PM

oh, thanks a lot.

floitsch 11/24/2022 03:05 PM

I think you can just set it to "", in which case you get a random one.

floitsch 11/24/2022 03:05 PM

However, then you need to set the clean_session flag.

floitsch 11/24/2022 03:05 PM

Having distinct client_ids is also a good approach.

floitsch 11/24/2022 03:06 PM

(Especially for adafruit, which keeps the information for each session).

floitsch

However, then you need to set the clean_session flag.

Shadi 11/24/2022 03:07 PM

at the end of program?

floitsch 11/24/2022 03:07 PM

No. The clean_session would be a flag to the SessionOption.

floitsch 11/24/2022 03:08 PM

But I think it's easier if you just use different client IDs.

floitsch 11/24/2022 03:08 PM

However, I recommend to reuse the same client, if you can.

floitsch 11/24/2022 03:08 PM

Having three TLS connections is almost impossible on these small boards.

floitsch 11/24/2022 03:08 PM

(The underlying mbedtls library needs a lot of memory).

Shadi 11/24/2022 03:09 PM

yes, I get this "MALLOC_FAILED" error

floitsch 11/24/2022 03:09 PM

Yeah... That's mbedtls.

floitsch 11/24/2022 03:09 PM

We are working on improving this, but it's not that easy.

floitsch 11/24/2022 03:09 PM

If you can, use a single mqtt client.

floitsch 11/24/2022 03:10 PM

If you made three different containers (one for each sensor), then you would need a fourth container with the mqtt and use RPC to talk to it.

floitsch 11/24/2022 03:10 PM

It's probably easier to merge the three containers. At least as a starting point.(edited)

Shadi 11/24/2022 03:12 PM

I think about it but I was wondering how can I publish the data from contaier_1 related to sensor_1 to this fourth container

floitsch 11/24/2022 03:12 PM

Yes. That's the challenge.

floitsch 11/24/2022 03:12 PM

There is an RPC protocol, but I would start by merging the three containers first.

floitsch 11/24/2022 03:12 PM

Just simpler.

floitsch 11/24/2022 03:13 PM

something like:

import .sensor1 as sensor1
import .sensor2 as sensor2
import .sensor3 as sensor3
import mqtt

main:
  // Create mqtt client.
  client := ..
  sensor1.main_with_client client
  sensor2.main_with_client client
  sensor3.main_with_client client

Shadi 11/24/2022 03:13 PM

merging containers as like is :
import app1
import app2
import app3

main:
app1_task := task:: app1.main
app2_task := task:: app2.main
app3_task := task:: app3.main

.

floitsch 11/24/2022 03:13 PM

Exactly.

floitsch 11/24/2022 03:14 PM

Yours is actually better, as it has the task (which I forgot).(edited)

Shadi 11/24/2022 03:14 PM

you gave me this hint previously

floitsch 11/24/2022 03:14 PM

Shadi 11/24/2022 03:14 PM

Thanks, I will try this

floitsch 11/24/2022 03:14 PM

We can then go and separate them again later.

Shadi 11/24/2022 03:15 PM

ok, I will test it now

floitsch 11/24/2022 03:15 PM

Just to give an idea of what will be needed:
https://github.com/toitlang/toit/blob/master/tests/services_simple_test.toit

toit/services_simple_test.toit at master · toitlang/toit

Program your microcontrollers in a fast and robust high-level language. - toit/services_simple_test.toit at master · toitlang/toit

floitsch 11/24/2022 03:16 PM

You will want to create a service (like the SimpleService), which takes data, and then pushes it to the mqtt broker.
The test just does a print in the log function. You would do a client.publish instead.

floitsch 11/24/2022 03:17 PM

Also: in the test we use spawn to launch a second app. You wouldn't need to do that, as your different containers would already run.

Shadi 11/24/2022 03:19 PM

I am a bit confused, I will test the approach now and will back with the result

Shadi 11/24/2022 03:57 PM

import dht11
import gpio
import mqtt
import net
import certificate_roots

PIN_NUMBER_1 ::= 25
PIN_NUMBER_2 ::= 32
ADAFRUIT_IO_USERNAME ::= ""
temperature1 :=25
temperature2 :=35

ADAFRUIT_IO_KEY ::= "**"
ADAFRUIT_IO_FEEDNAME ::= "toit_2"
ADAFRUIT_IO_FEEDNAME2 ::= "toit_1"

HOST ::= "io.adafruit.com"

interface SimpleService:

main:
adafruit_1
adafruit_2

adafruit_1:
sensor1 := dht11.Dht11 (gpio.Pin PIN_NUMBER_1)
exception := catch:
temperature1 =sensor1.read
print temperature1
if exception:
print "Error: $exception"
temperature1 =250

adafruit_2:
sensor2 := dht11.Dht11 (gpio.Pin PIN_NUMBER_2)
exception := catch:
temperature2 =sensor2.read
print temperature2
if exception:
print "Error: $exception"
temperature2 =250

class SimpleServiceDefinition implements SimpleService:
constructor:
network := net.open
transport := mqtt.TcpTransport.tls network --host=HOST
--root_certificates=[ certificate_roots.DIGICERT_GLOBAL_ROOT_CA ]
client := mqtt.Client --transport=transport
options := mqtt.SessionOptions
--client_id = "toit-example-client"
--username = ADAFRUIT_IO_USERNAME
--password = ADAFRUIT_IO_KEY
client.start --options=options
print "Connected to broker"
topic := "$ADAFRUIT_IO_USERNAME/feeds/$ADAFRUIT_IO_FEEDNAME"
topic2 := "$ADAFRUIT_IO_USERNAME/feeds/$ADAFRUIT_IO_FEEDNAME2"

client.publish topic "$temperature1".to_byte_array
client.publish topic2 "$temperature2".to_byte_array

Shadi 11/24/2022 03:58 PM

referring to your hints, I tried but I could not figure out how can I correct it

floitsch 11/24/2022 03:59 PM

You need the service definition only if you want to run your sensors in different containers.

Shadi 11/24/2022 04:00 PM

I tried to import my services in the fourth container but I could not import them and I got an error that *** package is not defined

floitsch 11/24/2022 04:00 PM

Are you now running Jaguar, or still Toit v1?

Shadi 11/24/2022 04:00 PM

now, I am trying with Toit v1.

floitsch 11/24/2022 04:00 PM

Toit v1 doesn't have the service feature yet.

floitsch 11/24/2022 04:01 PM

Give me a minute.

floitsch 11/24/2022 04:02 PM

I will write a small program that should (mostly) do what you need.

Shadi 11/24/2022 04:02 PM

Thanks a lot

floitsch 11/24/2022 04:09 PM

import dht11
import gpio
import mqtt
import net
import certificate_roots

PIN_NUMBER_1 ::= 25
ADAFRUIT_IO_FEEDNAME ::= "toit_2"

PIN_NUMBER_2 ::= 32
ADAFRUIT_IO_FEEDNAME_2 ::= "toit_1"

ADAFRUIT_IO_USERNAME ::= "your_username"
ADAFRUIT_IO_KEY ::= "your_key"

HOST ::= "io.adafruit.com"

main:
  network := net.open
  transport := mqtt.TcpTransport.tls network
      --host=HOST
      --root_certificates=[ certificate_roots.DIGICERT_GLOBAL_ROOT_CA ]
  client := mqtt.Client --transport=transport
  options := mqtt.SessionOptions
      --client_id = "toit-example-client"  // <== feel free to use a more descriptive name.
      --username = ADAFRUIT_IO_USERNAME
      --password = ADAFRUIT_IO_KEY
  client.start --options=options
  print "Connected to the broker"

  sensor1 := dht11.Dht11 (gpio.Pin PIN_NUMBER_1)
  topic1 := "$ADAFRUIT_IO_USERNAME/feeds/$ADAFRUIT_IO_FEEDNAME"

  sensor2 := dht11.Dht11 (gpio.Pin PIN_NUMBER_2)
  topic2 := "$ADAFRUIT_IO_USERNAME/feeds/$ADAFRUIT_IO_FEEDNAME_2"

  while true:
    th1 := sensor1.read
    print "Temperature 1: $th1.temperature"
    print "Humidity 1: $th1.humidity"
    client.publish topic1 "$th1.temperature".to_byte_array

    th2 := sensor2.read
    print "Temperature 2: $th2.temperature"
    print "Humidity 2: $th2.humidity"
    client.publish topic2 "$th2.temperature".to_byte_array

    // Make sure to have some delay between sending messages.
    // Adafruit only accepts one message every 2 seconds (and we
    // are sending them in pairs).
    sleep --ms=5_000

(edited)

floitsch 11/24/2022 04:10 PM

This is the simple approach without tasks, or services.

floitsch 11/24/2022 04:11 PM

This still uses the dht11 package and not the dhtxx package.

floitsch 11/24/2022 04:11 PM

It would be easy to switch to the dhtxx package once you are using v2.

Shadi 11/24/2022 04:13 PM

Thanks a lot; one question, If I want to use tasks or services, I must use jaguar, ja? my main goal is to measure power consumtion, so its important for me o have different Toit containers and see the total power consumption

floitsch 11/24/2022 04:13 PM

I see.

floitsch 11/24/2022 04:13 PM

You can use tasks, but not services.

floitsch 11/24/2022 04:14 PM

If you are worried about power consumption, then the sleep won't really help.

floitsch 11/24/2022 04:14 PM

If each sensor should run at the same time, then having them in one program is fine.

floitsch 11/24/2022 04:14 PM

You wouldn't win, if you put them into separate containers.

floitsch 11/24/2022 04:15 PM

Generally, the biggest power consumption is the WiFi. So you want to keep that at a minimum.

OPShadi

merging containers as like is : import app1 import app2 import app3 main: app1_task := task:: app1.main app2_task := task:: app2.main app3_task := task:: app3.main .

Shadi 11/24/2022 04:15 PM

Forexample here, app1 referes to my toit container for sensor one, right? but I could not import it in toit.io version

floitsch 11/24/2022 04:15 PM

Is there any reason you want to stay on v1?

Shadi 11/24/2022 04:16 PM

I tried with jaguar but I constantly got an error for installing it. I install required packets but when I try I get en error

floitsch 11/24/2022 04:16 PM

The app1.main refers to the import app1.
It should have probably been import .app1 as app1 where app1.toit would be the program app1.

OPShadi

I tried with jaguar but I constantly got an error for installing it. I install required packets but when I try I get en error

floitsch 11/24/2022 04:16 PM

Which kind of error?

floitsch 11/24/2022 04:16 PM

When flashing?

Shadi 11/24/2022 04:17 PM

yes

floitsch 11/24/2022 04:17 PM

Running?

Shadi 11/24/2022 04:17 PM

no, flashing

floitsch 11/24/2022 04:17 PM

You mean jag flash or jag run ?

Shadi 11/24/2022 04:17 PM

jag flash

floitsch 11/24/2022 04:17 PM

I would be curious which error you are getting.

floitsch 11/24/2022 04:18 PM

Assuming your USB cable is correct jag flash should just write a firmware onto your device.

Shadi 11/24/2022 04:20 PM

I thought this is related to my system and the packages I installed, I am now trying with another system

floitsch 11/24/2022 04:21 PM

Normally, jag should download everything it needs.

floitsch 11/24/2022 04:21 PM

(that's what it does when you write jag setup)

Shadi 11/24/2022 04:36 PM

the problem has been changed: Hard resetting via RTS pin...

Shadi 11/24/2022 04:36 PM

I tried to flash it but it remains

floitsch 11/24/2022 04:36 PM

Can you show me the error message?

floitsch 11/24/2022 04:37 PM

Hard resetting via RTS pin isn't necessarily fatal, depending on when it happens.

Shadi 11/24/2022 04:39 PM

Flashing device over serial on port '/dev/cu.SLAB_USBtoUART' ...
esptool.py v3.3.2-dev
Serial port /dev/cu.SLAB_USBtoUART
Connecting.....
Chip is ESP32-D0WD-V3 (revision 3)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 34:94:54:b8:03:08
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 8MB
Flash will be erased from 0x00001000 to 0x00004fff...
Flash will be erased from 0x00008000 to 0x00008fff...
Flash will be erased from 0x0000d000 to 0x0000efff...
Flash will be erased from 0x00010000 to 0x00172fff...
Flash params set to 0x0230
Compressed 15888 bytes to 11151...
Wrote 15888 bytes (11151 compressed) at 0x00001000 in 0.4 seconds (effective 323.8 kbit/s)...
Hash of data verified.
Compressed 3072 bytes to 165...
Wrote 3072 bytes (165 compressed) at 0x00008000 in 0.1 seconds (effective 353.0 kbit/s)...
Hash of data verified.
Compressed 8192 bytes to 31...
Wrote 8192 bytes (31 compressed) at 0x0000d000 in 0.1 seconds (effective 500.0 kbit/s)...
Hash of data verified.
Compressed 1452720 bytes to 889165...
Wrote 1452720 bytes (889165 compressed) at 0x00010000 in 13.7 seconds (effective 850.5 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

floitsch 11/24/2022 04:40 PM

That's good.

floitsch 11/24/2022 04:40 PM

What does jag monitor show?

floitsch 11/24/2022 04:40 PM

If the resetting via RTS pin doesn't work, just write jag monitor and then press the reset button.

Shadi 11/24/2022 04:43 PM

Starting serial monitor of port '/dev/cu.SLAB_USBtoUART' ...
ets Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0030,len:184
load:0x40078000,len:12700
ho 0 tail 12 room 4
load:0x40080400,len:2916
entry 0x400805c4
[toit] INFO: starting <v2.0.0-alpha.41>
[toit] DEBUG: clearing RTC memory: invalid checksum
[toit] INFO: using SPIRAM for heap metadata.
[wifi] DEBUG: connecting
[wifi] DEBUG: connected
[wifi] INFO: network address dynamically assigned through dhcp {ip: 10.10.155.156}
[jaguar] INFO: running Jaguar device 'green-bid' (id: 'e4a55073-d2e5-4ed3-b6d5-f698043b69b7') on 'http://10.10.155.156:9000'

floitsch 11/24/2022 04:43 PM

That looks like it's working.

floitsch 11/24/2022 04:43 PM

The device is named "green-bid"(edited)

Shadi 11/24/2022 04:44 PM

but when I try to run the code I get this:

Shadi 11/24/2022 04:44 PM

Error: didn't find any Jaguar devices

floitsch 11/24/2022 04:44 PM

I see.
Try to write jag scan 10.10.155.156

floitsch 11/24/2022 04:45 PM

Jaguar uses UDP broadcast packages to find devices.
It could be that your router doesn't let them through to your computer.

floitsch 11/24/2022 04:45 PM

(Alternatively, there is maybe a virus scanner, or so, that blocks them).

floitsch 11/24/2022 04:45 PM

If you write jag scan ... you tell Jaguar that you know where the device is, and that it shouldn't look for these packets.

floitsch 11/24/2022 04:45 PM

After that jag run ... should work.

Shadi 11/24/2022 04:47 PM

yeees, It works. Thanks a lot

floitsch 11/24/2022 04:47 PM

The v2 is significantly newer, and better in many areas.

Shadi 11/24/2022 04:48 PM

Now I can have several containers and merge them in another conatiner ?

floitsch 11/24/2022 04:48 PM

Let's take a step back, and figure out, whether you actually need that.

floitsch 11/24/2022 04:48 PM

Why do you want the individual containers?

Shadi 11/24/2022 04:50 PM

In the topic that I am working energy consumption is very important, so I want to show, If I can find the problem in sensors or services and after that, with the help of toit I can stop the corresponding service, In the long time, it is very beneficial and improve energy efficiency

floitsch 11/24/2022 04:51 PM

Ok. Yes... In that case working with containers is important

floitsch 11/24/2022 04:51 PM

Let me think a bit of how to solve this the best way.

Shadi 11/24/2022 04:52 PM

Thanks a lot. Your tips are very helpful

floitsch 11/24/2022 04:53 PM

The difficulty here is, that the network connection will drown out all the remaining energy consumption.

floitsch 11/24/2022 04:53 PM

The easiest would clearly be to have a container that runs the communication between the device and the MQTT broker, and let the individual sensors publish their data through that container.

floitsch 11/24/2022 04:54 PM

However, that would mean that starting and stopping the individual sensor containers wouldn't really do anything, as the real powerhungry process (the mqtt client) isn't affected by turning on/off a sensor.

Shadi 11/24/2022 04:58 PM

true, But imagine we have a motion sensor that constantly is measuring If there is a problem and for the whole time, this sensor is measuring the wrong data and if I can not stop it, a lot of energy will be wasted. but with toit, after detecting a problem I can stop it asap

floitsch 11/24/2022 05:01 PM

I have to admit, that some of what you are trying to do would be easier with Toit v1...
Since Jaguar is open-source, you can make it do what you want, but it's a bit harder than with v1.

floitsch 11/24/2022 05:01 PM

That said; even on v1 it's not completely straight-forward to do what you need to do.

Shadi 11/24/2022 05:02 PM

I see

floitsch 11/24/2022 05:03 PM

I will discuss this tomorrow with my coworkers. Let's see if they have ideas.

Shadi 11/24/2022 05:04 PM

I am very appreciative of your consulting

Shadi 11/25/2022 03:32 PM

Hi, I want to put my esp32 into deep sleep mode and I found with this command " esp32.deep_sleep (Duration --s=3604)" I can do this, but I could not "import esp32"

floitsch 11/25/2022 03:32 PM

esp32

library is only for v2 (Jaguar).

floitsch 11/25/2022 03:32 PM

Btw. I spoke with my colleagues.

floitsch 11/25/2022 03:33 PM

We have an example where the mqtt client lives in a separate process and is started on demand whenever another container wants to send data.

floitsch 11/25/2022 03:33 PM

That would probably be relatively similar to what you need.

Shadi 11/25/2022 03:34 PM

Exactly

floitsch 11/25/2022 03:35 PM

I just asked him if we have this example somewhere publicly accessible.

floitsch 11/25/2022 03:35 PM

As soon as he answers I will let you know.

floitsch 11/25/2022 03:35 PM

Getting back to the deep-sleep.

floitsch 11/25/2022 03:35 PM

On v1 you don't need any call to deep sleep, as the device automatically does that as soon as no application is running anymore.

Shadi 11/25/2022 03:35 PM

I ma using jaguar now

floitsch 11/25/2022 03:36 PM

Ok. Then you should be able to import esp32.

floitsch 11/25/2022 03:37 PM

If the editor shows an error, it might still be using toit for the syntax highlighting and analysis.

floitsch 11/25/2022 03:37 PM

It should use Jaguar if it detects both (toit and jag), though.

floitsch 11/25/2022 03:38 PM

This repository uses datacake to upload data.
https://github.com/kasperl/toit-datacake

GitHub - kasperl/toit-datacake: Datacake connector for the ESP32 ba...

Datacake connector for the ESP32 based on the Toit language. - GitHub - kasperl/toit-datacake: Datacake connector for the ESP32 based on the Toit language.

Shadi 11/25/2022 03:38 PM

I think my problem is related to this that jaguar can not find my device now

floitsch 11/25/2022 03:39 PM

You would just need to change the mqtt client from datacake to adafruit.

floitsch

You would just need to change the mqtt client from datacake to adafruit.

Shadi 11/25/2022 03:39 PM

very interesting. I will test it by tomorrow

OPShadi

I think my problem is related to this that jaguar can not find my device now

floitsch 11/25/2022 03:39 PM

If Jaguar can't see your device, then it should tell you.
An error message that import esp32 doesn't work is a different error.

floitsch

If Jaguar can't see your device, then it should tell you. An error message that import esp32 doesn't work is a different error.

Shadi 11/25/2022 03:41 PM

Yes, I just switch to jaguar, before that I use toit.io and now because of that I am getting this error. but from your yesterday hints, I will solve it and try to put it in deep sleep mode. thanks a alot

floitsch 11/25/2022 03:42 PM

Where do you get the error? and when?(edited)

Shadi 11/25/2022 03:44 PM

solved, I flash my esp completely and now jag can reqognize my device and deep_sleep also working,

149 messages in total