Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create tutorial for agent-agent interactions #515

Closed
2 of 3 tasks
tegefaulkes opened this issue Mar 20, 2023 · 27 comments
Closed
2 of 3 tasks

Create tutorial for agent-agent interactions #515

tegefaulkes opened this issue Mar 20, 2023 · 27 comments
Assignees
Labels
development Standard development r&d:polykey:core activity 3 Peer to Peer Federated Hierarchy

Comments

@tegefaulkes
Copy link
Contributor

tegefaulkes commented Mar 20, 2023

Specification

With Mercury's interest in Polykey, we need to create a guide for how to use Polykey to ultimately pull or clone a vault. During this process we should asses usability of the process. We can also test functionality of agent-agent interaction across networks.

All testing of these features so far have been on the same loopback device. Ideally we can test this across machines on the same local network. As well as machines between networks. I don't foresee these cases not working but it's good to be sure.

But for the tutorial, we need to show the following examples.

  1. Starting 2 agents with static ports.
  2. Adding an agent's connection information to one of the agents to seed the nodeGraph.
  3. Having the nodes ping each other.
  4. Having the nodes trust each other.
  5. setting up a vault on one of the nodes.
  6. Cloning a vault from the other node
  7. pulling changes from a vault

Stretch goals

  1. Setting up a private root node for forming a network
  2. Having nodes discover each other within the private network
  3. Explaining the limitations of said network. For example, the network is ad-hoc and not strict. In terms of discover-ability, two networks will implicitly join together if they share any node connections.

Additional context

Tasks

  1. Create a blog style tutorial detailing how to use Polykey in agent-agent interactions.
  2. Test that the examples work. Even better if we can verify local network communication and separate network communication
  3. Detail how a private ad-hoc network can still form with discovery even if nat-busting doesn't fully function currently. This is detailed by the stretch goals.
@tegefaulkes tegefaulkes added the development Standard development label Mar 20, 2023
@tegefaulkes tegefaulkes self-assigned this Mar 20, 2023
@tegefaulkes
Copy link
Contributor Author

tegefaulkes commented Mar 20, 2023

I was commenting about this on Issue #514. This is my mistake, i'll move the comments here.

@tegefaulkes
Copy link
Contributor Author

nodes ping can be used to ping a node if you provide the NodeId. This requires the node to already be within the node graph.

I'd expect a feature where you can specify the NodeId, host and port that you want to ping. This is useful in two ways.

  1. You can verify that a host and port is in fact the node you expected.
  2. You can check for nodes before adding them to the node graph. Pinging a node this way would add the node to the node graph. This can be a shortcut for adding nodes and checking they're online at the same time.

@tegefaulkes
Copy link
Contributor Author

There seems to be a bug with the nodes getall command. I'll have to check it out later.

[nix-shell:~/matixWorkspace/gitRepos/Polykey]$ npm run polykey -- nodes getall --node-path tmp/nodeA

> [email protected] polykey
> ts-node src/bin/polykey.ts "nodes" "getall" "--node-path" "tmp/nodeA"

ErrorPolykeyRemote: Remote error from RPC call - Cannot read properties of undefined (reading 'host')
  cause: TypeError: Cannot read properties of undefined (reading 'host')

@tegefaulkes
Copy link
Contributor Author

I've written a first blush of the guide. I'll comment it here in parts.

@tegefaulkes
Copy link
Contributor Author

Setting up nodes

For two agents to communicate with each other we need two agents to be running. For this example we're going to run two Polykey nodes on the same machine and have them communicate over the loop-back device.

First we want to start the first node. Since we are going to run two nodes on the same machine, we need to keep them separate. For this we need to override the node-path to a directory we want the Polykey node to be created in. for this example we're going to use a temp directory tmp. We're also going to set a static port value for the proxy.

Run the following command

npm run polykey -- agent start --node-path tmp/nodeA --proxy-port 55551

If this is the first time creating the node you should be prompted for the password twice. After that some information about the node should be printed.

✔ Enter new password … ********  
✔ Confirm new password … ********  
pid     476567  
nodeId  "vfjqk9v06k6p1tnuc4444irt0krasjmi1btmmf7ptaqg477em9390"  
clientHost      "127.0.0.1"  
clientPort      50617  
agentHost       "127.0.0.1"  
agentPort       35629  
proxyHost       "0.0.0.0"  
proxyPort       55551  
forwardHost     "127.0.0.1"  
forwardPort     38331  
recoveryCode    "drill video walnut message stone exhibit render snack comic maze deposit decline bless unit crater correct stool remain legend problem egg lecture approve normal"  

Note the NodeId here as vvrijg034ie10dnn05mv0b2lfo1g7nhv6kb63c03lh7qcc4eqo79g. This will differ for your node.

Now create a 2nd node using the same method, this time with the node-path of tmp/nodeB and port 55552. You should see the same password prompt and status read-out. For our example the 2nd node has a NodeId of vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg.

@tegefaulkes
Copy link
Contributor Author

Connecting and trusting nodes

Normally when a Polykey node is started, the first thing it does is automatically contact a seed node to join the network. In the absence of seed nodes to act as an entry point for a network. We we can tell a node how to find other nodes directly.

First we need to tell NodeA how to contact NodeB. For this we need to know the NodeId, host and port of NodeB. If we read the status output of NodeB when we started it up, we can see the following information.

nodeId  "vvrijg034ie10dnn05mv0b2lfo1g7nhv6kb63c03lh7qcc4eqo79g" 
...
proxyHost       "0.0.0.0"  
proxyPort       55551  

Using this we give NodeA NodeB's information using the following command.

npm run polykey -- nodes add --node-path tmp/nodeA vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg 127.0.0.1 55552

This will add the NodeB's connection information into NodeA's node graph. Now whenever we refer to NodeB via it's NodeId of vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg, NodeA will know where to connect to. Note that this step is only needed because both nodes do not share a network. If they did share a network then they could query nodes within the network to find each other.

To verify if this information is correct and a connection can be made, we can ping NodeB using the following command.

npm run polykey -- nodes ping --node-path tmp/nodeA vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg

If a connection can be made then you should get Node is Active. as the response. Otherwise you will get No response received.

When a connection is made between two nodes then they learn about each other. So when NodeA pinged NodeB, NodeB learned how to connect to NodeA. You can verify this by pinging NodeA from NodeB.

npm run polykey -- nodes ping --node-path tmp/nodeB vfjqk9v06k6p1tnuc4444irt0krasjmi1btmmf7ptaqg477em9390

Node is Active.

Now if NodeA wants to do anything more complex than pinging NodeB, we need to make NodeB trust NodeA. We can do this with the trust command. This will allow NodeB to receive notifications from NodeA, otherwise NodeB will just ignore them.

npm run polykey -- identities trust --node-path tmp/nodeB vfjqk9v06k6p1tnuc4444irt0krasjmi1btmmf7ptaqg477em9390

We should also have NodeA trust NodeB.

npm run polykey -- identities trust --node-path tmp/nodeB vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg

@tegefaulkes
Copy link
Contributor Author

Sharing vaults

Creating a Vault

A core feature of Polykey is sharing secrets between nodes. This is done by sharing the vaults that contain the secrets. In order to do this we need a vault to share. Start by creating a vault on NodeB. We can do this with the vaults create command, doing so will return a unique VaultId to identify that vault.

npm run polykey -- vaults create --node-path tmp/nodeB someVault

Vault zRMeFutQmJErPNR5rAE1LmN created successfully

Vaults and generally be referenced by it's name, here it's someVault. Or it's VaultId, here as zRMeFutQmJErPNR5rAE1LmN. An empty vault Is not very useful so let's add some data to it.

echo "this is a secret" > tmp/someSecret

npm run polykey -- secrets create --node-path tmp/nodeB tmp/someSecret someVault:someSecret

We should be able to see the secret in the vault now.

npm run polykey -- secrets list --node-path tmp/nodeB someVault

someSecret

Sharing a Vault

The next step is allowing NodeA to clone a vault from NodeB. This is done with the vaults share command. This will give NodeA permission to clone and pull this vault and send a notification to NodeA.

npm run polykey -- vaults share --node-path tmp/nodeB someVault vfjqk9v06k6p1tnuc4444irt0krasjmi1btmmf7ptaqg477em9390

NodeA should be able to clone the vault now. This is done with the vaults clone command.

npm run polykey -- vaults clone --node-path tmp/nodeA someVault vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg

If this completed with no errors, we should be able to see the vault in NodeA now.

npm run polykey -- vaults list --node-path tmp/nodeA

someVault:              zEsnWCGvgBqSuuu8r2SscSQ

We can access the secret as well.

npm run polykey -- secrets list --node-path tmp/nodeA someVault

someSecret

npm run polykey -- secrets get --node-path tmp/nodeA someVault:someSecret

this is a secret

Pulling vault changes

Vaults can be updated. If we wanted these changes then we can pull the vault. If a vault was cloned from another node then it keeps track of where it came from. So pulling a vault is pretty simple.

First we need to add a new secret to NodeB's vault.

npm run polykey -- secrets create --node-path tmp/nodeB tmp/newSecret someVault:newSecret

npm run polykey -- secrets list --node-path tmp/nodeB someVault

newSecret  
someSecret

Now we can tell NodeA to pull the changes from NodeB with Vaults pull.

npm run polykey -- vaults pull --node-path tmp/nodeA someVault

And see the changes

npm run polykey -- secrets list --node-path tmp/nodeA someVault

newSecret  
someSecret

npm run polykey -- secrets get --node-path tmp/nodeA someVault:newSecret

This is a new secret

@CMCDragonkai
Copy link
Member

This is all on the RPC migration branch right? So we just have to finish review and get that merged.

@tegefaulkes
Copy link
Contributor Author

Yep. I'll get client migration done today.

@tegefaulkes
Copy link
Contributor Author

There may be a bug with cloning a vault and then immediately accessing it.

[nix-shell:~/matixWorkspace/gitRepos/Polykey]$ npm run polykey -- secrets list --node-path tmp/nodeA someVault

> [email protected] polykey
> ts-node src/bin/polykey.ts "secrets" "list" "--node-path" "tmp/nodeA" "someVault"

ErrorPolykeyRemote: Remote error from RPC call - Cannot read properties of undefined (reading 'isLocked')
  command       vaultsSecretsList
  nodeId        v0000000000000000000000000000000000000000000000000000
  host
  port  0
  timestamp     Fri Mar 24 2023 14:50:04 GMT+1100 (Australian Eastern Daylight Time)
  cause: TypeError: Cannot read properties of undefined (reading 'isLocked')

Otherwise, just restarting the agent and trying the command again works.

I'll look into the problem.

@CMCDragonkai
Copy link
Member

Is this a regression?

@CMCDragonkai
Copy link
Member

Regression caused by RPC migration?

@tegefaulkes
Copy link
Contributor Author

I don't think so, The error is coming from inside the handler. I won't know for sure until I dive into it.

@tegefaulkes
Copy link
Contributor Author

The error is coming from the js-async-init CreateDestroyStartStop.ts @ready decorator

TypeError: Cannot read properties of undefined (reading 'isLocked')
    at VaultInternal.readF (/home/faulkes/matixWorkspace/gitRepos/Polykey/node_modules/@matrixai/async-init/src/CreateDestroyStartStop.ts:189:30)
    at Object.listSecrets (/home/faulkes/matixWorkspace/gitRepos/Polykey/src/vaults/VaultOps.ts:254:22)
    at f (/home/faulkes/matixWorkspace/gitRepos/Polykey/src/client/handlers/vaultsSecretsList.ts:37:39)
    at /home/faulkes/matixWorkspace/gitRepos/Polykey/src/vaults/VaultManager.ts:1012:14
    at async withF (/home/faulkes/matixWorkspace/gitRepos/Polykey/node_modules/@matrixai/resources/src/utils.ts:24:12)
    at async constructor_.withVaults (/home/faulkes/matixWorkspace/gitRepos/Polykey/src/vaults/VaultManager.ts:1005:12)
    at async /home/faulkes/matixWorkspace/gitRepos/Polykey/src/client/handlers/vaultsSecretsList.ts:33:14
    at async withF (/home/faulkes/matixWorkspace/gitRepos/Polykey/node_modules/@matrixai/resources/src/utils.ts:24:12)
    at async VaultsSecretsListHandler.handle (/home/faulkes/matixWorkspace/gitRepos/Polykey/src/client/handlers/vaultsSecretsList.ts:22:21)
    at async wrapperDuplex (/home/faulkes/matixWorkspace/gitRepos/Polykey/src/rpc/RPCServer.ts:324:9)

It's pretty odd. Even weirder that it only happens after cloning the vault and not after restarting the agent. It's likely due to the different methods of constructing the VaultInternal that is the root cause.

Cloning a vault constructs the VaultInternal with the cloneVaultInternal method. After restarting and accessing it, it would be constructed via the usual createVaultInternal method.

I'll compare the two to see if any differences in creation could be causing it.

@tegefaulkes
Copy link
Contributor Author

Yeah, I don't know about this one. Reviewing how a vault is created when accessed vs cloned, even looking at how the VaultManager handles it. Noting leaps out at me that could be the cause.

I checked and there is a test where we clone and immediately access a vault. and it runs fine. So it's not something that was missed in testing.

Given the fact that the error is related to life-cycle decorators. It seems possible It could be another problem relating to SWC and decorators.

@tegefaulkes
Copy link
Contributor Author

The bug does not happen when using the node dist/bin/polykey.js to run the agent. So it's a problem with using npm run polykey.

@CMCDragonkai
Copy link
Member

CMCDragonkai commented Mar 24, 2023 via email

@tegefaulkes
Copy link
Contributor Author

I'll have to investigate the differences between code compiled with SWC and TSC and possibly post an upstream issue.

For now the tutorial works fine if you use node dist/bin/polykey.js or restart the agent after cloning a vault.

@CMCDragonkai
Copy link
Member

Hey @tegefaulkes you said that it's not a problem when using node dist/bin/polykey.ts.

Can you compile your guide in the above 3 comments into a single one that can be used on the staging branch without using ts-node. We can solve the ts-node issue in a different issue.

I just want this to be ready to send to Mercury.

In the previous demo, we created some diagrams @amydevs, where did we post these? I forgot and cannot find them, as well as the commands to be executed. I know that we eventually turn that into the demo CLI gif on the README.md, but I want something like that for this part of the interaction.

@tegefaulkes
Copy link
Contributor Author

I can combine the guide in to a single comment and update it. Is there a better place to put it besides the comments here? It's pretty bulky already.

For now I'll put it in a comment with a details block.

@CMCDragonkai
Copy link
Member

It should go into the wiki. @amydevs where did you put all the demo materials last time?

@amydevs
Copy link
Contributor

amydevs commented Mar 27, 2023

@CMCDragonkai previous demo material was in this pr: #504

@tegefaulkes
Copy link
Contributor Author

nodes agent interaction tutorial

This tutorial assumes that you are working in the Polykey project's directory and you have successfully built it using npm run build.

Setting up nodes

For two agents to communicate with each other we need two agents to be running. For this example we're going to run two Polykey nodes on the same machine and have them communicate over the loop-back device.

First we want to start the first node. Since we are going to run two nodes on the same machine, we need to keep them separate. For this we need to override the node-path to a directory we want the Polykey node to be created in. for this example we're going to use a temp directory tmp. We're also going to set a static port value for the proxy.

Run the following command

node dist/bin/polykey agent start --node-path tmp/nodeA --proxy-port 55551

If this is the first time creating the node you should be prompted for the password twice. After that some information about the node should be printed.

✔ Enter new password … ********  
✔ Confirm new password … ********  
pid     476567  
nodeId  "vfjqk9v06k6p1tnuc4444irt0krasjmi1btmmf7ptaqg477em9390"  
clientHost      "127.0.0.1"  
clientPort      50617  
agentHost       "127.0.0.1"  
agentPort       35629  
proxyHost       "0.0.0.0"  
proxyPort       55551  
forwardHost     "127.0.0.1"  
forwardPort     38331  
recoveryCode    "drill video walnut message stone exhibit render snack comic maze deposit decline bless unit crater correct stool remain legend problem egg lecture approve normal"  

Note the NodeId here as vvrijg034ie10dnn05mv0b2lfo1g7nhv6kb63c03lh7qcc4eqo79g. This will differ for your node.

Now create a 2nd node using the same method, this time with the node-path of tmp/nodeB and port 55552. You should see the same password prompt and status read-out. For our example the 2nd node has a NodeId of vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg.

Connecting and trusting nodes

Normally when a Polykey node is started, the first thing it does is automatically contact a seed node to join the network. In the absence of seed nodes to act as an entry point for a network. We we can tell a node how to find other nodes directly.

First we need to tell NodeA how to contact NodeB. For this we need to know the NodeId, host and port of NodeB. If we read the status output of NodeB when we started it up, we can see the following information.

nodeId  "vvrijg034ie10dnn05mv0b2lfo1g7nhv6kb63c03lh7qcc4eqo79g" 
...
proxyHost       "0.0.0.0"  
proxyPort       55551  

Using this we give NodeA NodeB's information using the following command.

node dist/bin/polykey nodes add --node-path tmp/nodeA vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg 127.0.0.1 55552

This will add the NodeB's connection information into NodeA's node graph. Now whenever we refer to NodeB via it's NodeId of vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg, NodeA will know where to connect to. Note that this step is only needed because both nodes do not share a network. If they did share a network then they could query nodes within the network to find each other.

To verify if this information is correct and a connection can be made, we can ping NodeB using the following command.

node dist/bin/polykey nodes ping --node-path tmp/nodeA vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg

If a connection can be made then you should get Node is Active. as the response. Otherwise you will get No response received.

When a connection is made between two nodes then they learn about each other. So when NodeA pinged NodeB, NodeB learned how to connect to NodeA. You can verify this by pinging NodeA from NodeB.

node dist/bin/polykey nodes ping --node-path tmp/nodeB vfjqk9v06k6p1tnuc4444irt0krasjmi1btmmf7ptaqg477em9390

Node is Active.

Now if NodeA wants to do anything more complex than pinging NodeB, we need to make NodeB trust NodeA. We can do this with the trust command. This will allow NodeB to receive notifications from NodeA, otherwise NodeB will just ignore them.

node dist/bin/polykey identities trust --node-path tmp/nodeB vfjqk9v06k6p1tnuc4444irt0krasjmi1btmmf7ptaqg477em9390

We should also have NodeA trust NodeB.

node dist/bin/polykey identities trust --node-path tmp/nodeB vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg

Sharing vaults

Creating a Vault

A core feature of Polykey is sharing secrets between nodes. This is done by sharing the vaults that contain the secrets. In order to do this we need a vault to share. Start by creating a vault on NodeB. We can do this with the vaults create command, doing so will return a unique VaultId to identify that vault.

node dist/bin/polykey vaults create --node-path tmp/nodeB someVault

Vault zRMeFutQmJErPNR5rAE1LmN created successfully

Vaults and generally be referenced by it's name, here it's someVault. Or it's VaultId, here as zRMeFutQmJErPNR5rAE1LmN. An empty vault Is not very useful so let's add some data to it.

echo "this is a secret" > tmp/someSecret

node dist/bin/polykey secrets create --node-path tmp/nodeB tmp/someSecret someVault:someSecret

We should be able to see the secret in the vault now.

node dist/bin/polykey secrets list --node-path tmp/nodeB someVault

someSecret

Sharing a Vault

The next step is allowing NodeA to clone a vault from NodeB. This is done with the vaults share command. This will give NodeA permission to clone and pull this vault and send a notification to NodeA.

node dist/bin/polykey vaults share --node-path tmp/nodeB someVault vfjqk9v06k6p1tnuc4444irt0krasjmi1btmmf7ptaqg477em9390

NodeA should be able to clone the vault now. This is done with the vaults clone command.

node dist/bin/polykey vaults clone --node-path tmp/nodeA someVault vd2nmd13qqj718ivke8n8si6cgdsd6ufgche84moofr2fhh9vj2jg

If this completed with no errors, we should be able to see the vault in NodeA now.

node dist/bin/polykey vaults list --node-path tmp/nodeA

someVault:              zEsnWCGvgBqSuuu8r2SscSQ

We can access the secret as well.

node dist/bin/polykey secrets list --node-path tmp/nodeA someVault

someSecret

node dist/bin/polykey secrets get --node-path tmp/nodeA someVault:someSecret

this is a secret

Pulling vault changes

Vaults can be updated. If we wanted these changes then we can pull the vault. If a vault was cloned from another node then it keeps track of where it came from. So pulling a vault is pretty simple.

First we need to add a new secret to NodeB's vault.

node dist/bin/polykey secrets create --node-path tmp/nodeB tmp/newSecret someVault:newSecret

node dist/bin/polykey secrets list --node-path tmp/nodeB someVault

newSecret  
someSecret

Now we can tell NodeA to pull the changes from NodeB with Vaults pull.

node dist/bin/polykey vaults pull --node-path tmp/nodeA someVault

And see the changes

node dist/bin/polykey secrets list --node-path tmp/nodeA someVault

newSecret  
someSecret

node dist/bin/polykey secrets get --node-path tmp/nodeA someVault:newSecret

This is a new secret

@tegefaulkes
Copy link
Contributor Author

@amydevs can you run through this and test it?

@tegefaulkes
Copy link
Contributor Author

Created a PR for adding this to the docs at

MatrixAI/Polykey-Docs#19

@CMCDragonkai
Copy link
Member

This is done. https://polykey.io/docs/tutorials/vault-sharing.

Closing this.

@tegefaulkes can you create a big issue for the problem of vault cloning in ts-node?

@tegefaulkes
Copy link
Contributor Author

Sure thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
development Standard development r&d:polykey:core activity 3 Peer to Peer Federated Hierarchy
Development

No branches or pull requests

3 participants