Dedicated methods for DNS-SD, CoRE Link-Format discovery, and other approaches?

[JKRhb]

Experimenting with both DNS-SD and discovery via the CoRE Link-Format in dart_wot, I got the impression that it could make sense to add dedicated discovery methods for both approaches and maybe even replace the generic discover method in this context.

As the two approaches belong to the introduction phase, the new methods could be used solely for discovering URLs pointing to Things, which could be wrapped in objects containing a url and a type (such as Thing or Directory). The results could then be followed up with either the requestThingDescription or the exploreDirectory method. As multiple incoming responses are possible when using multicast with both approaches, the two new methods could once again return an AsyncIterable which would enable an asynchronous processing of the results.

I think both approaches (and especially the case of using DNS-SD with multicast DNS) can be motivated via use cases quite well. However, I wanted to create an issue in this regard already since it also concerns a general design question if adding new methods to the WoT namespace would be the way to go here or if we risk making the API surface a bit too large. Is there maybe also a concept of sub-namespaces in WebIDL?

Another question we could also discuss in this regard concerns the generic discover method. Should we keep it as it is right now or should we maybe also replace it with more specialized methods in the future (like discoverNearbyThings or discoverThingsViaBluetooth)? From what I've seen so far when it comes to discovery with Bluetooth and NFC, I suppose that the return type of such methods could also be a URI or an AsyncIterable of URIs, right?

At some point, we might want to ask the same questions regarding DID – generally speaking, I think there is the question of which methods defined in the Discovery spec we should support directly and which should be delegated to "userland".

[danielpeintner]

Another question we could also discuss in this regard concerns the generic discover method. Should we keep it as it is right now or should we maybe also replace it with more specialized methods in the future (like discoverNearbyThings or discoverThingsViaBluetooth)?

If we want to introduce more specialized methods I would rather use a configuration option that can be passed in in some way so that if we want to add yet another way of discovering the option needs to be updated and not yet a new method needs to be created.

[JKRhb]

If we want to introduce more specialized methods I would rather use a configuration option that can be passed in in some way so that if we want to add yet another way of discovering the option needs to be updated and not yet a new method needs to be created.

Yeah, that sounds reasonable, I think. Maybe this approach could actually also be used for DNS-SD and CoRE Link-Format after all. However, I've got the feeling that we should consider changing the return type to URLs instead of Thing Descriptions, so that we use the discover method for modeling the introduction phase and the other methods for the exploration phase.

What could the shape of the configuration option look like? Would it make sense to have something like

{
  "method": "dns-sd",
  "domainName": "_wot._tcp.example.org.",
}  

where the method field could be used to distinguish the different approaches while the additional fields specify the parameters of the respective method? If we take this route we could also consider defining a registry for adding new methods in the future.

[zolkis]

There have been two design approaches:

• single one-shot discover() method with options (including local, broadcast, directory, direct, and any) that will produce TDs only.
• and dedicated separate functions that may return links, directories, and then TDs.
  It was assumed the runtime had been provisioned for discovery.

I think it makes sense to separate the methods for introduction phase (that return links or directories), which are a new thing since the discovery TF started working. Then, discovery will (continue to) return TDs.

The API design should consider following:

• the assumption is that we have a single entity handling discovery, e.g. the runtime
• so that we can set options/constraints on the discovery process, making 2-phase discovery possible (which is stateful)
• in which case we should separate the methods for introduction phase and discovery phase, because they return different stuff
• but if stateless combined-phase discovery is desired by the app, it should be still possible, i.e. we should specify default constraints. Note this is also possible by a local function, so we should just keep in mind this use case: if it's easy to do with the same API, it's good, otherwise it's trivial to combine 2 phases in one local function.

We could contain discovery related functionality in its own namespace, or object. It could contain the following:

• set discovery method (by function or by option) and handle the returned links / directories
• set domain constraints or give a URL (argument or option)
• start discovery and handle the returned TDs
• signal stopping discovery (after which the runtime suppresses further hits).

I like stateless interactions more, so I'd prefer the approach via options (with default values), and hiding processing the directories/links by the runtime, providing only the TDs to the app.

But if we want a control point in the apps, then we need an additional method, that handles the introductory phase, and the returned stuff can be used as options to the second phase, as above.

In the end, we just need an additional introductory phase method, named e.g. explore(), that returns URLs (either links or directories encapsulated as a map) that can be fed to the discover() method as options.

[relu91]

Call 13/05:
We should find a decision forward. @zolkis In second-screen the discovery API is a method to hide configuration details (i.e. to avoid finger printing). The use case is getting TDs, we already supporting it with the two functions. If anything we should come up with a concrete proposal with API definition and algorithms. This is really related to the use-case "find things around me".

[JKRhb]

I've been experimenting a bit with different approaches for the discover method in dart_wot and, as suggested by @zolkis, I now arrived at a solution where the discovery parameters can be set via the underlying platform (a Servient class) by passing in a list of discovery configuration objects. When invoking the discover method, the current configurations are then passed on internally to the ThingDiscoveryProcess which can then be used to asynchronously iterate over the discovery results.

My current approach now looks something like the example below. Here, the Servient is configured to only use one discovery configuration that first obtains URLs via mDNS and then retrieves the TDs via HTTP.

  final servient = Servient.create(
    clientFactories: [
      HttpClientFactory(),
    ],
    discoveryConfigurations: [
      // Use DNS-SD with mDNS for discovery
      DnsSdDConfiguration(
        protocolType: ProtocolType.tcp,
        discoveryType: DiscoveryType.thing,
      ),
    ],
  );

  final wot = await servient.start();
  
  await for (final thingDescription in wot.discover()) {
    // Handle discovered Thing Description
  }

I think that is actually a quite simple solution that also enables you to quickly perform a rediscovery of TDs and that can also be extended with new methods in the future. As mentioned in #551, this also corresponds pretty much with what is already specified in the Scripting API. The only downside might be that, if you want to perform different discovery procedures, you first need to reconfigure the Servient and then re-invoke the discover method, which is not exactly ideal.

So the question I had here was if we want to make it possible to override the default configuration defined in the underlying platform via a set of “well-known” configurations or if we should at least create some sort of extension point (i.e., an object for options that is passed to the discover method) that could be used by implementations for overriding.

[lu-zero]

Wouldn't be better having a separate object that deal with the discovery lifecycle and pass it to a function that returns an iterable of TDs ? e.g a wot.discoverVia({mDNS, CoRE, FlatList})

[zolkis]

@JKRhb
Do we need to tie discovery parameters with the servient, i.e. why the limitation that we can have only one discovery config at a time at the servient?

Using a discovery API that is passed discovery options, an app could specify multiple discovery options, the implementations could span a discovery process (session) for each type of options separately, and collect the results for the app.

In addition, when an app would start another additional discovery process with different (or overlapping) discovery options, the implementation could break it down to suitable discovery transactions/processes and manage those behind the scenes, just providing the results to the app.

So I would decouple the API / SW interface of doing discovery from the app, from the actual implementations that needs to comply with the Discovery spec, and map the app requests to suitable discovery transactions/processes.

This design decision would work when there is 1:1 mapping between discovery API and discovery process, and also when there is something to encapsulate (the discovery processes in the background may change more often, but the API should not change that often).

[JKRhb]

Hmm, I think I somehow misunderstood you, @zolkis, and for some reason thought that discover should not accept any configuration options. My preferred approach would actually look somewhat similar to the one posted above by @lu-zero:

const discoveryOptions = [
  {
    "type": "dns-sd",
    ...
  },
  {
    "type": "core",
    ...
  },
];

// Variant 1 (in accordance with the current API design)
const discoveryProcess = await wot.discover({ discoveryOptions });

for await (const thingDescription of discoveryProcess) {
  // Handle thingDescription
}


// Variant 2 (without awaiting a discoveryProcess)
for await (const thingDescription of wot.discover({ discoveryOptions })) {
  // Handle thingDescription
}

This approach might be a bit simpler than instantiating separate objects for the discovery lifecycle, as things would be handled behind the scenes according to the configurations that have been passed in. However, this way you would probably need to specify the available options or provide a way for the user to find out which configurations are supported by the platform.

[lu-zero]

Most of the discovery processes can be one-off or long-running, so having an object that can be at least started, stopped and queried separately might be good.

[JKRhb]

Most of the discovery processes can be one-off or long-running, so having an object that can be at least started, stopped and queried separately might be good.

Hmm, yeah, that is a good point. The alternative would be to provide a timeout parameter in the configuration so that a discovery process would be cancelled automatically after some time.

I suppose we could also consider having both kinds of APIs, so a set of “built-in” discovery methods (that do not require the instantiation of a separate object for discovery), and an interface that would also allow for the addition of custom/adjusted discovery mechanisms. Using only one approach would also work for me, though.

[zolkis]

Right, so I think we are back to something like the old discovery API:

• a discover() method takes an options parameter and returns a Promise that resolves with a control object (representing the discovery process/session) - and fails when the options are not supported
• the control object can be used for iterating over the results and also handle non-critical errors; we have explored a few alternatives here, and also in what to include in the options (a default timeout parameter could be one of them).

The main question was how to map 2-phase discovery to the API.
In theory, a separate method would be a good start, if we want to allow apps to control the 1st phase.
But we must define well what is the output (and input parameters) of the 1st stage, and how does the 2nd stage use that (and differ from that).
When that is clarified, we can decide the optimal API shape(s) for one-shot discovery vs 2-phase discovery.