Apply baseURI to list of things generated by HTTP server

[justus237]

Setting the baseURI in the configuration of an HTTP server currently changes the forms for all interaction affordances, but does not apply to the list of Things exposed by the server (https://github.com/eclipse/thingweb.node-wot/blob/master/packages/binding-http/src/http-server.ts#L694).

Not sure whether this is intended or not, but because the baseURI configuration is separate from base in the Thing Description, I thought it might be worth having baseURI applied consistently. This also avoids leaking local IP addresses if a node-wot server is exposed to the Internet through port forwarding, which was my use case.

If this is worth fixing/merging, I'll open a pull request.

[danielpeintner]

@justus237 your thinking makes sense to me but maybe I miss some (corner) cases.

@relu91 do you know?

[relu91]

Yes we definitely need a fix for this issue, since I always have hard time to use node-wot inside a container runtime. Maybe, it is too ambitious but if @justus237 is willing to work with a PR we should come up with a mechanism that would work for all the protocol bindings. Currently, to personalize the form href generation we can:

1. Specify Helpers.setStaticAddress: this will work with all the protocol bindings but it only affects the host (i.e., you can't personalize the port) of the form href.
2. Use baseURI: you can effectively personalize everything that is before path, but it only works for the HTTP binding.

We need a single mechanism consistent across all binding templates. We also need something that can work for different use cases:

• Web Things are behind a proxy (i.e., I want to change scheme, host, port)
• Dynamic IPs but one single DNS (i.e. just want to change the host)

[relu91]

Going back to the issue, the list of available Things should be moved to the appropriate syntax ( see the discovery spec) but meanwhile we can at least fix it so that it can be really used.

[justus237]

Going back to the issue, the list of available Things should be moved to the appropriate syntax ( see the discovery spec) but meanwhile we can at least fix it so that it can be really used.

Trying to find this in the discovery spec, are you referring to https://w3c.github.io/wot-discovery/#exploration-directory-api-things-listing?

[justus237]

We need a single mechanism consistent across all binding templates.

Am I correct to assume that this would ("only") need to touch all the binding templates with servers, i.e., HTTP(S), CoAP, MQTT, (firestore?) and websocket? The most straightforward approach would be to extend the Helpers class?

[relu91]

Going back to the issue, the list of available Things should be moved to the appropriate syntax ( see the discovery spec) but meanwhile we can at least fix it so that it can be really used.

Trying to find this in the discovery spec, are you referring to https://w3c.github.io/wot-discovery/#exploration-directory-api-things-listing?

yes that one, but it would be nice to support CoRE Directories too.

Am I correct to assume that this would (only) need to touch all the binding templates with servers, i.e. HTTP(S), CoAP, MQTT, (firestore?) and websocket? The most straightforward approach would be to extend the Helpers class?

Yes exactly, currently we ask the helper class for a list of possible host addresses, but then each binding server appends its information. The tricky part here is that the Helper class should stay protocol agnostic, and single bindings should be allowed to personalize hrefs if no specific configuration is given.

[justus237]

Reviving this, there are currently already multiple competing "definitions" for specifying forms for Thing Descriptions.

You can define the base in the TD, in the AAS Asset Interface Description Submodel you define "EndpointMetadata" which appears to be base + contentType (endpoint is not clearly defined across protocols anyway, and especially not defined in any HTTP RFC... it is defined for web services only) and you would like to specify the "origin" (scheme + host + port) (origin is somewhat defined in some HTTP RFCs, but those are not Internet Standards AFAIK) for each form directly.

What is the big picture here? I feel like there are multiple competing mechanisms that all attempt to do the same thing, but they’re all different. Is the goal to have maximum flexibility and just have all the mechanisms co-exist?

With the list of available Things I can somewhat understand that CoRE directories are likely most useful for CoAP bindings, and putting all TDs in a list is probably most suited for the HTTP bindings. So all bindings might have a different "list of things" mechanism, but they do not compete with each other, whereas the origin/endpoint/base concepts all clash.

[relu91]

What is the big picture here? I feel like there are multiple competing mechanisms that all attempt to do the same thing, but they’re all different. Is the goal to have maximum flexibility and just have all the mechanisms co-exist?

You are right, there is a little bit of confusion here mostly because node-wot is an experimental effort to implement the specifications and during that time it tried to follow back all the features but sometimes we had to find some workaround. Let me try to clean up the waters a little bit.

There are two mechanisms to control the final href value in forms: generative (node-wot), and declarative (Thing description specification).

Generative

In node-wot, you can create your ExposedThings by defining an ExposeThingInit object that looks like a Thing Description or better a Partial Thing Description. The ExposeThingInit contains the structure of what will be the real TD. This structure is used as a hint and then the runtime should fill in all the missing information. We are trying to follow what is specified inside the Scripting API in the produce but we might miss some bits because sometimes we can't keep up with the evolution of the spec (if you have time to review the process there is also an open issue #911).

In practice, we don't allow the user to specify hrefs in the ExposeThingInit but we generate those ourselves using the variables and methods described above. Doing so we probably don't abide by the base field in the Thing Description (the declaritive mechanism) and we just create a full path ourselves.

Declarative

Thing Descriptions have a member called base which can be used to calculate the full href of a form when it has a relative URI. However, as you probably noticed, does not work well when you use multiple protocol bindings inside a single TD. This is an issue that we are going to solve in the next version of the W3C WoT specification. In node-wot, we could allow it for single binding TDs, but we should raise an error/warning when used in conjunction with other protocol bindings.

Does this help?

With the list of available Things I can somewhat understand that CoRE directories are likely most useful for CoAP bindings, and putting all TDs in a list is probably most suited for the HTTP bindings. So all bindings might have a different "list of things" mechanism, but they do not compete with each other, whereas the origin/endpoint/base concepts all clash.

Let's talk about this in a dedicated issue and focus this thread only on the generation of the base URI of ExposedThings. (#786 might be the right place).

[justus237]

Thanks for the reply and explanation. Is there a specific reason why base is not really used in node-wot? As far as I can see, the node-wot client would parse it correctly.

Going back to the implementation, assume I create a servient with an HttpServer to directly access interaction affordances but also have a locally running MQTT broker, with an MqttBrokerServer added to the servient, and I have both the (separately but locally hosted) broker and the HTTP server somehow proxied to the Internet.

If I want to customize the form URIs, I would want to specify the schema, host and port for each protocol binding, e.g. mapping each protocol to its TLS-secured version, meaning the Helpers has to keep track of which server has which schema, host and/or port set.

Another option is adding the baseUri concept to other protocol bindings, or introducing new config options that allow specifying both what is generated in the TD and the configuration for the server that is actually created.

It sounded like you would prefer using Helpers to ensure a consistent mechanism across protocol bindings?

[relu91]

Thanks for the reply and explanation. Is there a specific reason why base is not really used in node-wot? As far as I can see, the node-wot client would parse it correctly.

My guess:

1. The base concept is underspecified in the Thing Description specification and therefore is complex to come up with the correct implementation that works for all bindings
2. Complexity in introducing it in node-wot
3. Lack of volunteers

Other than these I don't think there is any blocking issue. We can get around point 1 by incubating a proposal here, but since in the spec base is just a string we can't do it in a backward-compatible way. A possible approach is to use base for HTTP and define a node:base/node:endpoint/node:servers term where we can use an object to map the right base for each protocol. The downside of this approach is that if the WoT WG decides to discard it we have to re-implement it with a different syntax or structure.

It sounded like you would prefer using Helpers to ensure a consistent mechanism across protocol bindings?

Yeah, this was my original idea, cause it would be quite elegant to have a single place that is used by all the bindings. But now I'm figuring out that different protocols will need different strings for schemes or other URI parts. Configuring that without leaking protocol infos into the core is not feasible.

Therefore, I think extending the configuration options of each protocol binding is the only good solution now. it just needs to be consistent across all of them.

[egekorkan]

I think that overall it is important to differentiate what features of the partial TD given to the produce() function are exposed at the end. As a rule of thumb, nothing about the protocols, content types and security can be given as a requirement on what node-wot should do. What TD can be produced depends very heavily on what the servient contains at the time the produce function is called. If there is no HTTP binding loaded to the servient, a base uri containing http:// means nothing. From my point of view, to have consistency, some TD terms should be prohibited or loudly ignored to be passed to the produce function. Anything that influences the following terms should be given as part of a config in the servient or in the binding or via a new configuration mechanism we can propose:

• forms
• base
• securityDefinitions (here I would exclude security since that can be actually chosen per affordance etc.)

[justus237]

The way the code is currently structured, the ExposedThing is not directly aware of all protocol binding servers (only of the servient, I think), and the servers are not aware of each other. Finding a common base and converting some hrefs to relative URIs would need to happen in the ExposedThing (presumably in expose()), by going through all forms.

Regarding the configuration at the servient, as far as I can tell, this is basically what Helpers is used for. We might get away with enabling the configuration of the authority part of an URI (host+port), without schema, which could also make the case of finding a common base easier.

Regardless, I think the "start" should be implementing the configuration at the protocol binding level. The main question I have here is, how "backwards" compatible node-wot is meant to be. Currently the CoAP server takes configuration parameters directly in the constructor, while the HTTP server takes HttpConfigs. Also, if a new mechanism (e.g., formHrefScheme, formHrefHost, formHrefPort) is introduced, the question is whether it should override baseUri in the HTTP server, or whether baseUri should be removed completely.

[danielpeintner]

Is there a specific reason why base is not really used in node-wot?

It turned out to be much simpler to do one a pass over the TD to convert all hrefs to absolute URIs. Doing so guarantees that passing a form object is enough. Otherwise, besides form we would need base and to

• construct over and over again a full URI by concatenating the base with relative URI
• base might not be needed

This is surely not a strong argument but at least a reason at that time... I think

[relu91]

Regardless, I think the "start" should be implementing the configuration at the protocol binding level. The main question I have here is, how "backwards" compatible node-wot is meant to be. Currently the CoAP server takes configuration parameters directly in the constructor, while the HTTP server takes HttpConfigs. Also, if a new mechanism (e.g., formHrefScheme, formHrefHost, formHrefPort) is introduced, the question is whether it should override baseUri in the HTTP server, or whether baseUri should be removed completely.

Yes, we can improve the approach later. I would say that what is important is to allow the user to configure the form generation as he pleased... we can refine the inner workings later. About backward compatibility, we are still in 0.x.y phase so we don't have that strict requirement to maintain everything backward compatible. Currently, we are increasing the minor only if core API changes, we have less care about protocol bindings APIs and configuration parameters. Therefore, about baseUri, I'd refactor it with the new approach.