How to serve HTML, CSS, Javascript files

Toit

#How to serve HTML, CSS, Javascript files

Thread channel in help

hhv0001 12/21/2023 05:26 PM

Hi, back in my Arduino days (two days ago) I had a webserver running on my ESP32-S2. It served one HTML file and a CSS file and Javascript files. There was also JSON going back and forth. The website was used for adjusting settings in my ESP software, for instance choose a colour for the attached LEDs.

I am struggling to find documentation on how to achieve the same with Toit. Including the HTML as a string in my Toit program is not only uncool but even inconvenient. What's the trick here? Do you know?

I read that people have SD-cards connected to their ESP32, but I do not have that and would like to avoid adding hardware to my PCB if possible.

I could get the files from an external webserver but I would rather not depend on that, it opens up a can of worms (i.e. proxies, firewalls, etc.) and I would like the device to be mostly self-sufficient.

Thank for any tips or thoughts on the matter.

bitphlipphar 12/21/2023 05:57 PM

Maybe something like 'assets' would work for you?

bitphlipphar 12/21/2023 05:58 PM

https://github.com/toitlang/toit#adding-container-assets

GitHub - toitlang/toit: Program your microcontrollers in a fast and...

Program your microcontrollers in a fast and robust high-level language. - GitHub - toitlang/toit: Program your microcontrollers in a fast and robust high-level language.

bitphlipphar 12/21/2023 05:58 PM

We'll probably need to write up how to do this easily through Jaguar.

hhv0001 12/21/2023 07:04 PM

Thanks Kasper, that will do. A bit more work than just "jag watch programname.toit", but by the looks of it, it can be easily automated. I'll give it a go.

bitphlipphar 12/21/2023 07:06 PM

@hhv0001

You're getting into the deep end. Love it

hhv0001 12/21/2023 07:09 PM

I see there's an --assets argument to "jag run" and "jag watch"! That's looking great. These assets do not change often.

hhv0001 12/22/2023 11:51 AM

So far I have this:

else if request.path == "/index.html":
      writer.headers.set "Content-Type" "text/html"
      writer.write assets.decode["index.html"]

and it works!

bitphlipphar 12/22/2023 11:53 AM

Awesome!

hhv0001 12/22/2023 12:30 PM

Btw I get this compile error when trying to close the ResponseWriter:
Class 'ResponseWriter' does not have any method 'close'
writer.close

Shall I make a bug report for that or is it my bad? (I'm not a OO-guru but shouldn't the class ResponseWriter have the class Writer as a parent, and not Object?)

hhv0001 12/22/2023 12:31 PM

I tried writer.close-write and writer.close-writer too...

bitphlipphar 12/22/2023 01:29 PM

@hhv0001

That sounds a bit weird. The ReponseWriter class does have a close method.

bitphlipphar 12/22/2023 01:30 PM

Which version of pkg-http are you using? You can probably tell by looking at your package.lock file.

bitphlipphar 12/22/2023 01:30 PM

https://github.com/toitlang/pkg-http/blob/main/examples/server.toit#L50

pkg-http/examples/server.toit at main · toitlang/pkg-http

Package: Implement your REST server or client in Toit and run it on your ESP32. - toitlang/pkg-http

hhv0001 12/22/2023 01:46 PM

It says :
packages:
pkg-http:
url: github.com/toitlang/pkg-http
name: http
version: 1.9.3
hash: 108c436cc990535f5d70c380ef68081c38840f4c

hhv0001 12/22/2023 01:46 PM

That server.toit is the example I copied as a starting point.

bitphlipphar 12/22/2023 01:47 PM

It looks like you're stuck on a slighty old version.

hhv0001 12/22/2023 01:47 PM

Ah.

bitphlipphar 12/22/2023 01:47 PM

Try jag pkg uninstall http and then reinstall it using jag pkg install http.(edited)

bitphlipphar 12/22/2023 01:50 PM

Not sure how you ended up with a 1.x version. When / how did you install this in the first place?

bitphlipphar 12/22/2023 01:50 PM

(maybe we have some outdated documentation somewhere)

hhv0001 12/22/2023 01:54 PM

I had to uninstall websocket as well and now I have version 2.3.4, no more errors. Thanks! I have probably installed that older version of http a few months ago, on my first Toit-attempt.

bitphlipphar 12/22/2023 01:58 PM

I think the websocket support is now included in the http package. Does that sound right @erikcorry?

erikcorry 12/22/2023 02:13 PM

Yes

erikcorry 12/22/2023 02:13 PM

Yes

bitphlipphar 12/22/2023 02:15 PM

Yes

hhv0001 12/22/2023 04:39 PM

Thanks guys.

bitphlipphar 12/22/2023 05:07 PM

Yes

bitphlipphar 12/22/2023 05:08 PM

Thanks for giving this a spin!

hhv0001 12/25/2023 09:44 AM

It's not terribly important for me atm but I am running into something I can't fix. I have started off with the webserver example and added assets as I mentioned above. That works reasonably well, except when I load a page that has links to CSS and Javascript in it, then the requests for these CSS and Javascript files take additional 30 seconds to download. I am also seeing this:

DEBUG: incoming request {peer: 10.1.0.166:56947, path: /index.html}
Handling request for /index.html

Decoding by jag, device has version <2.0.0-alpha.127>

EXCEPTION error.
Connection closed

hhv0001 12/25/2023 09:44 AM

0: tcp-write <sdk>/net/modules/tcp.toit:199:3
1: TcpSocket.write <sdk>/net/modules/tcp.toit:165:16
2: Writer.write <sdk>/writer.toit:39:23
3: ChunkedWriter.write-header <pkg:pkg-http>/chunked.toit

13
4: ChunkedWriter.write <pkg:pkg-http>/chunked.toit:77:5
5: ResponseWriter.write <pkg:pkg-http>/server.toit:202:18
6: main.<lambda> webserver.toit:122:14
7: Server.run-connection.<block> <pkg:pkg-http>/server.toit:164:19
8: catch.<block> <sdk>/core/exceptions.toit:124:10
9: catch <sdk>/core/exceptions.toit:122:1
10: catch <sdk>/core/exceptions.toit:85:10
11: Server.run-connection <pkg:pkg-http>/server.toit:163:9
12: Server.listen.<block>.<lambda>.<block>.<block> <pkg:pkg-http>/server.toit:113:26
13: catch.<block> <sdk>/core/exceptions.toit:124:10
14: catch <sdk>/core/exceptions.toit:122:1
15: catch <sdk>/core/exceptions.toit:97:10
16: Server.listen.<block>.<lambda>.<block> <pkg:pkg-http>/server.toit:112:18
17: Server.listen.<block>.<lambda> <pkg:pkg-http>/server.toit:109:38
18: Server.listen.<block> <pkg:pkg-http>/server.toit:129:37
19: Server.listen <pkg:pkg-http>/server.toit:85:3
20: main webserver.toit:111:10

hhv0001 12/25/2023 09:45 AM

INFO: Internal Server error - Connection closed {peer: 10.1.0.166:56947, path: /index.html}
DEBUG: connection ended {peer: 10.1.0.166:56947, reason: Connection closed}

and

Handling request for /add-color-wheel.js
DEBUG: connection ended {peer: 10.1.0.166:56939, reason: DEADLINE_EXCEEDED}

The trouble is not with the assets I think. When I replace looking up an asset with HTML or CSS I see the same behaviour.

Any pointers on how to fix this or examine this are welcome. I don't think I fully understand how server.listen in the main: function works f.i.

bitphlipphar 12/25/2023 10:14 AM

@hhv0001

You may benefit from allowing your http server to serve multiple requests concurrently. Pass --max-tasks=3 to the constructor of http.Server; see https://github.com/toitlang/pkg-http/blob/main/src/server.toit#L34.

pkg-http/src/server.toit at main · toitlang/pkg-http

Package: Implement your REST server or client in Toit and run it on your ESP32. - toitlang/pkg-http

bitphlipphar 12/25/2023 10:15 AM

(it could be that 3 isn't the perfect setting for you)

hhv0001 12/28/2023 10:24 AM

That helps, but unfortunately it postpones the problem. So the first three files download very fast and after that there's a 30 second delay. Increasing to 5 tasks leaves 1 file with a 30 second delay, increasing it to 6 tasks solves it. It seems a bit arbitrary, but I haven't looked at the source code though, there may be a good reason for that behaviour. Still, a 30 second delay...

bitphlipphar 12/28/2023 11:37 AM

Your browser may download multiple resources in parallel.

bitphlipphar 12/28/2023 11:38 AM

I think it is 6 for Chrome.

bitphlipphar 12/28/2023 11:38 AM

(connections per server)

bitphlipphar 12/28/2023 11:40 AM

Can't claim that I fully understand why you see the 30s delay though. Maybe @erikcorry knows?

hhv0001 12/28/2023 03:00 PM

Yes, the browser sending 6 requests at a time I can understand, but this 30 second delay... Ah, 30 seconds is the "--.read_timeout=DEFAULT_READ_TIMEOUT" parameter. And it is used in line 143 and further in server.toit:
while true:
request := null
with_timeout read_timeout:
request = connection.read_request

hhv0001 12/28/2023 03:02 PM

I can add a Duration parameter for a timeout of 3 seconds to the constructor of the http.Server and then the wait is 3 seconds.

erikcorry 12/29/2023 09:59 AM

The browser keeps trying to open connections instead of using the ones it already has open.
After leaving a connection unused for 30s, it times out and gets closed, and at that point we can accept another connection.

erikcorry 12/29/2023 10:02 AM

I'm not sure what the best solution is here.
Chrome is not expecting a server to have trouble with 6 parallel connecitons, but it's a lot of TCP connections for an ESP32.

erikcorry 12/29/2023 10:06 AM

Reducing the timeout to 3s would be an improvement, but it would be better if Chrome just reused the connection it already has instead of opening new ones.

hhv0001 12/29/2023 10:16 AM

I think it probably does. Safari also uses 6 connections AND sends a keep-alive header. I think 6 connections are used because for any ordinary website these days a lot of stuff needs to be downloaded. When I look at a regular news-site in the Netherlands I see 262 documents being downloaded. Not all from the same domain, but the main domain has 11 documents and the largest number of docs from one domain is +- 100!

hhv0001 12/29/2023 10:20 AM

There's room for improvement here on the part of the browsers I think. But people want fast loading websites, which is what you normally get when 6 connections are used by default.

erikcorry 12/29/2023 10:21 AM

It looks like Chrome only opens two connections for very simple websites.

erikcorry 12/29/2023 10:21 AM

If you add a few more assets it starts opening more.

erikcorry 12/29/2023 10:22 AM

For the example in pkg-dhs-simple-server (captive portal) it only opens two.

erikcorry 12/29/2023 10:22 AM

When I add more assets it opens six.

hhv0001 12/29/2023 10:24 AM

OK, I'll reduce my assets. I am not sure I can go as low as two :-). Thanks for your help @erikcorry !

erikcorry 12/29/2023 10:25 AM

That's not a great solution

erikcorry 12/29/2023 10:30 AM

I'll check if accepting the socket, then immediately closing it causes the browser to use the ones it already ahs.

erikcorry 12/29/2023 10:30 AM

has

hhv0001 12/29/2023 10:34 AM

It isn't a great solution, but an esp32 is a restricted environment, I can live with that. I'll give it a go. svg's can be included in the html, css and js too (highly unrecommended security wise, but ...). Hopefully XHR requests don't count for opening a new connection - but I could use a websocket then which is a much better solution.

erikcorry 12/29/2023 10:37 AM

Immediately closing the socket causes broken-image icons in the browser. It doesn't just fall back to the connection it already has open.

hhv0001 12/29/2023 10:42 AM

Thanks for checking. That was fast.

bitphlipphar 12/29/2023 10:43 AM

We just need HTTP/2 support

erikcorry 12/29/2023 10:52 AM

Heh

erikcorry 12/29/2023 10:53 AM

For a server that does not use HTTPS, 30s is a very long timeout on unused connections.
They can be set up much faster than that, so it's a bad tradeoff.

erikcorry 12/29/2023 10:53 AM

We should probably reduce that.

hhv0001 12/29/2023 11:10 AM

I've reduced the timeout to just --ms=10 and that works fine here with --max-tasks=1, so I'm good for now :-). Thanks guys.

erikcorry 12/29/2023 01:31 PM

That's agressive :-).

erikcorry 12/29/2023 01:35 PM

I'm still thinking about whether I can make a nicer fix here, but in the mean time we should use the latest HTTP library in the captive portal example https://github.com/toitlang/pkg-dns-simple-server/pull/9

Use newer HTTP library in example by erikcorry · Pull Request #9 · ...

65 messages in total