Websocket Client and Server for Client Service API

[tegefaulkes]

Specification

The client based communication is to be implemented using websockets. Currently we have a proof of concept using the node ws library. but a lot of the code needs to be formalised into a client and server class for managing the state. I will expand this spec when I come back to it, This is placeholder for now to track the task.

We need to create a client and server class for client communication.

Websocket Server

We need to create a Websocket server class for abstracting the logic and life-cycle of the Websocket server code. The class doesn't really hold state outside of itself so it doesn't require a destroy method. I think It can be made a StartStop? otherwise CreateDestroy should do.

The server needs to take a stream creation callback of (streamPair, connectionInfo) => void. This will be used to call the stream handler when a connection is established. We may have limited connection information on the connecting side. So far as I can tell the remote port can't be obtained from the connection. The client certificate isn't applicable here but I think we can obtain the remote IP address. For local information, the host and port are easily obtained.

The server needs to support secure connection over HTTPS/SSL and reject any insecure connections. We also need the ability to update the certificates at any time though that may not be possible. The uWebsocket library requires that we write the certs to disk to access them. For this we need to encrypt the files with a random password and pass that password to the server when loading them. I am unsure what method is used to encrypt them at this time.

Back-pressure propagation needs to be applied to both the readable and writable stream. The websocket has some signalling for when the buffer is filling up. for the writable stream the ws.send() returns a number for success, back-pressure and message dropped. We can also check how full the buffer is at any time. There is also a drain event for when the back-pressure has drained some. For the readable stream we need to use a BYOB readable stream and signal to the websocket back-pressure by corking it. I'm still a bit fuzzy on this.

Half open connection needs to be supported. if the stream handler closes the writable before the readable or vice versa. the websocket shouldn't be closed until both streams are closed. If the websocket closes abruptly then both streams are closed. If the writable is still used this will result in an error. it is up to the stream handler to deal with this.

The server needs to refuse any normal HTTP requests.

Websocket client

We need to create a Websocket client class for abstracting the logic of the client code. The class should extend either StartStop or CreateDestory depending how the server is implemented or other constraints.

Creation of the the client should take host, port address and NodeId of the target server. This should be used to establish the connection.

The client needs to connect over a secure connection using a wss:// url. We need to add custom logic for authenticating the server's certificate against the expected NodeId of the server we're connecting to. This can be done via the upgrade event.

Back-pressure needs to be signalled to the streams. I need to prototype how to do this with the ws library.

Half open connections need to be supported. The websocket should only be ended when both the readable and writable are closed. If both are closed early then this should result in the websocket ending. If the websocket ends early then both streams should be closed. If the writable is still used this will result in an error. it is up to the stream caller to deal with this.

Additional context

• Related Transport Agnostic RPC implementation #498
• Related discussion starting at Transport Agnostic RPC implementation #498 (comment)

Tasks

• 1. Create a ClientServer for websocket based communication.
  • Create a ClientServer class for handling life-cycle of the server.
  • Add Secure connection with TLS, use encryption when writing files and reading them with the server. 0.5 days - Differed to Encrypted private key PEM format #508
  • Apply back-pressure propagation to the streams. 0.5 days
  • Support half open connections by only closing the websocket when both streams have closed.
  • Handle abruptly dropped connections. 0.5 days
  • Refuse any non-websocket communication. trivial
  • Support IPv6
• 2. Create a ClientClient for websocket based communication.
  • Create a ClientClient class for handling life-cycle of the client. 0.5 days
  • Make the connection secure using TLS. 0.5 days
  • Add custom authentication logic for checking the server's certificate against a NodeId. 0.5 days
  • Apply back-pressure propagation to the streams. 1 days
  • Support half open connections by only closing the websocket when both streams have closed. 0.5 days
  • Handle abruptly dropped connections. 0.5 days
  • Connection timeout logic
  • Support IPv6

[CMCDragonkai]

The spec should include sections for WebSocketClient and WebSocketServer.

[CMCDragonkai]

And of course transition/differences with uWebSockets library since it can be portable for ios and android.

[CMCDragonkai]

This was not solved by #498.

[CMCDragonkai]

Websockets will the main client service server, but HTTP1/2 server can also be possible, but beware of the constraints mentioned in #166.

Now with transport agnostic RPC, we can figure out any kind of transport suitable for our RPC system.

Any HTTP API would probably look similar to graphql system. There's 1 endpoint and you just send POST requests all the time, with the JSON data sent in.

Do note to help debugging your JSON parser should accept \n delimited JSON messages too, not just immediately concatenated. I suggested that concatenated JSON should be our format, which actually allows for no separation, or arbitrary whitespace separation.

[tegefaulkes]

The uWebsocket server has some oddities when handling messages compared to the usual event driven way. The main one is that there is no message or data events on the websocket itself.

for example. Usually we'd handle the connection and then register events for data. E.G.

// Paraphrasing here.
const server = createServer()

server.on('open', (ws) => {
  ws.on('data', handleData) // Here we reigister an event for handling data for this specific websocket.
})

But the difference with uws is that the websocket doesn't have any data events. So handling the data is more in the context of the server and not the socket.

// Paraphrasing here.
const server = createServer()

server.ws('/*', {
  upgrade: (res, req, context) => {
    res.upgrade(
      // Setting custom data for this websocket
      {
        some: 'data',
      },
      req.getHeader('sec-websocket-key'),
      req.getHeader('sec-websocket-protocol'),
      req.getHeader('sec-websocket-extensions'),
      context,
    );
  },
  open: (ws) => {
    // Can get user data here
    const userData = ws.getUserData()
  },
  message: (ws, _message) => {
    // Can get user data here
    const userData = ws.getUserData();
    // We can use the userData for multiplexing
   // OR we can use `ws` as a key in an object map
  },
})

[tegefaulkes]

I'm having trouble getting the remote connection information with uws. It's not critical for client communication however.

[CMCDragonkai]

I'm having trouble getting the remote connection information with uws. It's not critical for client communication however.

What did you try? Also did you google/check the repo issues?

Sometimes things are not well documented and you need to check the source code.

[CMCDragonkai]

The uWebsocket server has some oddities when handling messages compared to the usual event driven way. The main one is that there is no message or data events on the websocket itself.

for example. Usually we'd handle the connection and then register events for data. E.G.

// Paraphrasing here.
const server = createServer()

server.on('open', (ws) => {
  ws.on('data', handleData) // Here we reigister an event for handling data for this specific websocket.
})

But the difference with uws is that the websocket doesn't have any data events. So handling the data is more in the context of the server and not the socket.

// Paraphrasing here.
const server = createServer()

server.ws('/*', {
  upgrade: (res, req, context) => {
    res.upgrade(
      // Setting custom data for this websocket
      {
        some: 'data',
      },
      req.getHeader('sec-websocket-key'),
      req.getHeader('sec-websocket-protocol'),
      req.getHeader('sec-websocket-extensions'),
      context,
    );
  },
  open: (ws) => {
    // Can get user data here
    const userData = ws.getUserData()
  },
  message: (ws, _message) => {
    // Can get user data here
    const userData = ws.getUserData();
    // We can use the userData for multiplexing
   // OR we can use `ws` as a key in an object map
  },
})

What happens when you don't upgrade?

[CMCDragonkai]

It's fine for the server to emit events directly.

But without a socket object, how do you close specific websocket connections?

[tegefaulkes]

Upgrade is used here just to add custom data to the websocket. I don;t think it's strictly needed except to set the user data to an object that I can modify later. If I don't add an upgrade handler it works as normal.

As for closing a connection there are a few ways.

• The listening socket can be closed with uWebsocket.us_listen_socket_close(listenSocket).
• A connection can be refused in upgrade with res.close() or res.end().
• A websocket can be closed with ws.close(), ws.end().

[tegefaulkes]

I don't think I can get the port of the connecting client.

uNetworking/uWebSockets#1034

the websocket has a getRemoteAddressAsText() and getRemoteAddress() methods but they just return the host and no port information.

[tegefaulkes]

For TLS on the server we need to provide the certificates paths so the underlying native code can load it. This means we can't provide the certificate as a string like we usually do.

Here are the options for creating the server

    key_file_name?: RecognizedString;
    cert_file_name?: RecognizedString;
    ca_file_name?: RecognizedString;
    passphrase?: RecognizedString;
    dh_params_file_name?: RecognizedString;
    ssl_ciphers?: RecognizedString;
    ssl_prefer_low_memory_usage?: boolean;

We just need to write the files and then provide the paths to the server. But this means we are writing the private key PEM to the file system just to use it.

[CMCDragonkai]

I don't think I can get the port of the connecting client.

uNetworking/uWebSockets#1034

the websocket has a getRemoteAddressAsText() and getRemoteAddress() methods but they just return the host and no port information.

The port they connected to you is what you would know. But the port you need to respond to... Should be knowable otherwise packets couldn't be sent back.

[CMCDragonkai]

For TLS on the server we need to provide the certificates paths so the underlying native code can load it. This means we can't provide the certificate as a string like we usually do.

Here are the options for creating the server

    key_file_name?: RecognizedString;
    cert_file_name?: RecognizedString;
    ca_file_name?: RecognizedString;
    passphrase?: RecognizedString;
    dh_params_file_name?: RecognizedString;
    ssl_ciphers?: RecognizedString;
    ssl_prefer_low_memory_usage?: boolean;

We just need to write the files and then provide the paths to the server. But this means we are writing the private key PEM to the file system just to use it.

So a similar situation with js-quic. The quiche library takes file paths for the certs.

However the quiche code also takes an SSL context structure. And I've yet to figure this all out, but it means it is possible to set the certificate in-memory.

Again for this websocket it's possible that the C++ code exposes how to load it. Which is why our own binding would be useful.

Now if it really doesn't work. The most secure way is to create a temporary file, you can encrypt it and pass the password for the UWS to load it. Alternatively if it could take a file descriptor, if you open the file, then delete it, you could refer to the FD. However this won't work in windows.

So most ideal is our own binding into the C++ to do this, then backup solution is to use encrypted temporary file. See the password option above.

[CMCDragonkai]

Upgrade is used here just to add custom data to the websocket. I don;t think it's strictly needed except to set the user data to an object that I can modify later. If I don't add an upgrade handler it works as normal.

As for closing a connection there are a few ways.

• The listening socket can be closed with uWebsocket.us_listen_socket_close(listenSocket).
• A connection can be refused in upgrade with res.close() or res.end().
• A websocket can be closed with ws.close(), ws.end().

What is the listenSocket? Is this the server itself?

I'm guessing you don't get a separate socket per websocket connection cause technically each websocket connection isn't a separate socket (not a separate TCP socket right?).

[CMCDragonkai]

I thought the upgrade would be for upgrading http to websockets. Like the 101 response code. Is it this https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Upgrade?

Also is uws http2 or http1.1?

[tegefaulkes]

The upgrade handler is called when upgrading to a websocket, yes.

I couldn't find anything about what version of HTTP it used. I saw in the issues that it supports HTTP3 experimentally. I'm not certain if it uses HTTP 2 or not. I'll look into it.

[CMCDragonkai]

That's why I was asking what happens if you don't upgrade, does it stay as a HTTP server?

[tegefaulkes]

Maybe I don't know what you mean by 'don't upgrade'? This could mean a few things.

• If you don't have an upgrade handler then the default behaviour of upgrading to a websocket happens.
• If you reject a connection in the upgrade handler then the connection is rejected.
• If you connect with a http request and not as a websocket then you are routed to one of the http handlers if you registered any. We can default to rejecting these with adding a handler server.any('/*', (res) => res.end('some message')). I haven't tested this yet.

[CMCDragonkai]

The HTTP handler can be useful for our agent status page. It should show HTML instead of JSON. Actually it could display HTML or JSON depending on the accept header. Auth is still needed for the status though. It could show less info if not authenticated.

[tegefaulkes]

Abruptly dropped connections means the underlying connection fails. This means the usual control signals for ending a socket are never sent such as the end frame. So far as I know this means any websocket on the other side can't tell the connection has dropped or actually ended.

To fix this we need to make use of the ping/pong handlers for detecting the connection's life. A connection failing is considered exceptional. So the readable and writable stream on both ends should throw an error in this case.

To properly test this I need to run the Clientserver or clientClient as a separate process and kill that process. This is something we have done before in network tests.

[CMCDragonkai]

So you don't get an error event on the websocket client/server if the connection is broken? It's possible this is the case unless you have a "keepalive" system. I think websocket has ping/pong specified in its specification. See the SO post here: https://stackoverflow.com/questions/23238319/websockets-ping-pong-why-not-tcp-keepalive

If we enable it, it should be available as part of the WS libraries we are using. No need to create our own version.

[CMCDragonkai]

Also: https://www.rfc-editor.org/rfc/rfc6455#section-5.5.2

[CMCDragonkai]

Regarding the encryption of the private key for web socket server.

In order to encrypt the private into an encrypted PEM file, we need to follow the PKCS#5 standard (version 2) to produce an encrypted PKCS#8 pem file.

For example, in openssl, it has a pkcs8 subcommand. https://www.openssl.org/docs/man1.0.2/man1/pkcs8.html

In this subcommand there is a -v2 alg option. And in the comments it says:

The alg argument is the encryption algorithm to use, valid values include des, des3 and rc2. It is recommended that des3 is used.

So this will basically end using an algorithm specified in pkcs#5 v2 RFC. However it's not clear which algorithm this is, and that it's OID is going to be.

One way to solve this is to actually use the openssl pkcs8 command, and run it with a plaintext PKCS8 key. Using the -v2 des3 option, and you get back your PKCS8 encrypted PEM.

Subsequently you then just need to interrogate this file. Probably using another openssl command.

Actually in the updated version https://www.openssl.org/docs/man1.1.1/man1/openssl-pkcs8.html, it turns out the -v2 alg can be omitted, and the default cipher becomes AES256GCM.

So that means, we can just do the encryption using openssl pkcs8 command, then open up the file, and parse it, to get the algorithm identifier.

This identifier is going to be an OID. The OID is just a string that looks like 1.3.101.110. But we just need to know which one is for this one.

Use the ASN1 parser to see if you can open up the encrypted file and just console.log out the version identifier string.

Once you get the confirmed OID, we can actually proceed with building this encrypted pem file.

To do this, use https://github.com/PeculiarVentures/asn1-schema/blob/master/packages/pkcs8/src/encrypted_private_key_info.ts.

Then construct the EncryptedPrivateKeyInfo object, with the algorithm identifier. Because this identifier may not exist in any of asn1 libraries. The string can be defined in the utils if we need to do this. It should go into the keys/utils.

The next thing to do is to actually encrypt it. It is explained here.

https://datatracker.ietf.org/doc/html/rfc5208#section-6

We would reuse our own existing code:

  const pkcs8 = new asn1Pkcs8.PrivateKeyInfo({
    privateKeyAlgorithm: new asn1X509.AlgorithmIdentifier({
      algorithm: x509.idEd25519,
    }),
    privateKey: new asn1Pkcs8.PrivateKey(
      new asn1.OctetString(privateKey).toASN().toBER(),
    ),
  });

That provides us the private key info object.

The encryption process involves the following two steps:

 1. The private-key information is BER encoded, yielding an octet
    string.

 2. The result of step 1 is encrypted with the secret key to give
    an octet string, the result of the encryption process.

But what we actually need is something like:

new asn1.OctetString(pkcs8).toASN().toBER()

That then produces an arraybuffer.

Then that array buffer has to go through the encryption process of AES-256-GCM, whatever is specified as part of the OID we discover through openssl pkcs8.

The resulting function can be called privateKeyToPEMEncrypted. Then to be symmetric privateKeyFromPEMEncrypted.

Good thing is that our libsodium supports aes256gcm. So we should be able re-use symmetric algos. We currently have not exposed this from the sodium-native JS wrapper, so you'll need to have a look at that.

[CMCDragonkai]

I just looked at https://github.com/sodium-friends/sodium-native/blob/master/binding.c, it doesn't export the aes256gcm from libsodium. So the only alternative right now is the peculiar webcrypto which would have it.

However we need to create a new feature issue to replace webcrypto with whatever is possible in libsodium.

[CMCDragonkai]

Did you know that it's possible in Python to create temporary files that don't actually exist on disk. It relies on the O_TMP option in Linux, other OSes may have something similar. I'm not sure if this is exposed to NodeJS at all atm, but that's another option to have something that is in-memory, but just so happens to have a file handler. Since the underlying library requires a file path, you actually have to pass a "path" to the file descriptor. Unfortunately this is not portable, since paths to the process's file descriptor is very OS-specific.

[CMCDragonkai]

Forget about intermediate solutions like that. We would eventually want to memlock anything that is sensitive in-memory anyway.