Custom JSON serialization of TipSetKey for array-of-CIDs

[rvagg]

🤷 is this OK to do here?

Basically: the byte form of a TipSetKey isn't very useful as a consumer of the Lotus API, we don't give it like that anywhere. So a user would have to decoded the base64 bytes, extract the CIDs from it using one of the CID libraries, then re-form it back into the array-of-CIDs dag-json form to post it back to a Lotus API to do anything with it (like calling ChainGetTipSet).

Before:

$ curl -s -X POST -H "Content-Type: application/json" --data '{"method":"Filecoin.F3GetLatestCertificate","params":[],"id":2,"jsonrpc":"2.0"}' http://127.0.0.1:1235/rpc/v1
{"id":2,"jsonrpc":"2.0","result":{"GPBFTInstance":42292,"ECChain":[{"Epoch":2175657,"Key":"AXGg5AIgSRY8LQrdggB5IOtuEPM7t3QVCZsQd6X6w6ZGM6e0AE4BcaDkAiA3gbWY6IFo8bRTc+9TKGlmCEU+bgNQzpoNVhhEr/ySswFxoOQCICxs2tKsVpIAiqdqZXLp4aIOQBFyBOW+XElIC0NFkVZ5","PowerTable":{"/":"bafy2bzaceabae7y7mjaib2z4obmvicmqbbpuhihnhdomzjwcghwal536y3oeg"},"Commitments":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"Epoch":2175658,"Key":"AXGg5AIgG52GGH6mEskkX8woDYC2ddur/wak+PurcR9q2E3d6a8=","PowerTable":{"/":"bafy2bzaceabae7y7mjaib2z4obmvicmqbbpuhihnhdomzjwcghwal536y3oeg"},"Commitments":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]}],"SupplementalData":{"Commitments":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PowerTable":{"/":"bafy2bzaceabae7y7mjaib2z4obmvicmqbbpuhihnhdomzjwcghwal536y3oeg"}},"Signers":[0,2,1,1,1,1],"Signature":"kEmVhxBHXh8kE77j58KtOgMzC0srFf1TPULigMCy3ipMcPHycHpWX0ANsYEVKmpiEyxmGRge79XQiFHb9rMiXU44i8Ts/VFZ2vQuHkXXExrDt0PnejnJ8Oh95niq+GYz","PowerTableDelta":null}}

After:

$ curl -s -X POST -H "Content-Type: application/json" --data '{"method":"Filecoin.F3GetLatestCertificate","params":[],"id":2,"jsonrpc":"2.0
"}' http://127.0.0.1:1235/rpc/v1
{"id":2,"jsonrpc":"2.0","result":{"GPBFTInstance":42376,"ECChain":[{"Key":[{"/":"bafy2bzaceco7323ltaq3efz3godq5gfzpdnt664hx32brfzpk2bxhnioywoyi"},{"/":"bafy2bzacedjaquoof5sj5zgqa5zwnd2u5rbrcxu4xgl62diti2u3n7opq4ozk"},{"/":"bafy2bzacebhdoylimje4ep4ymf5k6vy7taaexe65n4cvm4gcml6my65pby634"},{"/":"bafy2bzacecgv6xsvnnhvhbucarhocdbwtjzlnrdcqilz2tvwz2ad2tygfk44w"},{"/":"bafy2bzacebfiituy7hzuwc43czd35eal4q2wp75fm6uoyp2dkr2vcmq4j7yrs"}],"Epoch":2175744,"PowerTable":{"/":"bafy2bzaceabae7y7mjaib2z4obmvicmqbbpuhihnhdomzjwcghwal536y3oeg"},"Commitments":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},{"Key":[{"/":"bafy2bzacednxacmhjsthqhd4y42pvy3wyow5ai32xlhgg5kn6cm6wqc7dkfx4"},{"/":"bafy2bzaced65livizlohxbwgzk5r2zxtuofj2v2yzzpfmut7cibcj7quwxmbq"},{"/":"bafy2bzaceatetc6pat6535o5sqguvf3izbdoajilsslibfv67su3ywpg2odhw"}],"Epoch":2175745,"PowerTable":{"/":"bafy2bzaceabae7y7mjaib2z4obmvicmqbbpuhihnhdomzjwcghwal536y3oeg"},"Commitments":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]}],"SupplementalData":{"Commitments":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],"PowerTable":{"/":"bafy2bzaceabae7y7mjaib2z4obmvicmqbbpuhihnhdomzjwcghwal536y3oeg"}},"Signers":[0,2,1,1,1,1],"Signature":"gbyZNJBy4bkuDJDcYqDw0woZI0FK748w1f4nC6LmTyCadwAZRKzV2gTevWzHS7YcB6VytkOQQE5RKlPxTOm/B7bF0xIN5gR/uHXrHCWTOX6ganIgqYog8g5n0c1df4ya","PowerTableDelta":null}}

(different epoch but you get the idea)

But now I can at least feed that tsk back into lotus:

$ curl -s -X POST -H "Content-Type: application/json" --data '{"method":"Filecoin.ChainGetTipSet","params":[[{"/":"bafy2bzacednxacmhjsthqhd4y42pvy3wyow5ai32xlhgg5kn6cm6wqc7dkfx4"},{"/":"bafy2bzaced65livizlohxbwgzk5r2zxtuofj2v2yzzpfmut7cibcj7quwxmbq"},{"/":"bafy2bzaceatetc6pat6535o5sqguvf3izbdoajilsslibfv67su3ywpg2odhw"}]],"id":2,"jsonrpc":"2.0"}' http://127.0.0.1:1235/rpc/v1

An alternative would be to not expose certs.FinalityCertificate directly in Lotus but to wrap it in a new struct that uses the native Lotus TipSetKey: https://github.com/filecoin-project/lotus/blob/5791b4a9786bc259df2c1cc2775fd9aaf62d25fb/api/api_full.go#L984-L987

[rvagg]

It is a bit annoying on the Go side in Lotus that this will round-trip just fine but we still end up with a non-standard TipSetKey to deal with, so we'll need to cast it into a Lotus TipSetKey before it can be used elsewhere. That argues for replacing certs.FinalityCertificate with a Lotus version in the API. But if we do that, is there F3 client code (used by miners?) that needs to consume this object so would end up having to transform it back into the F3 form to use it?

[codecov]

Codecov Report

Attention: Patch coverage is 84.21053% with 9 lines in your changes missing coverage. Please review.

Project coverage is 69.56%. Comparing base (c2f99cf) to head (5c25653).

Files with missing lines	Patch %	Lines
gpbft/chain.go	85.36%	4 Missing and 2 partials ⚠️
gpbft/gpbft.go	81.25%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #756      +/-   ##
==========================================
- Coverage   69.68%   69.56%   -0.12%     
==========================================
  Files          74       74              
  Lines        7494     7551      +57     
==========================================
+ Hits         5222     5253      +31     
- Misses       1863     1884      +21     
- Partials      409      414       +5     

Files with missing lines	Coverage Δ
certs/certs.go	92.68% <ø> (ø)
gpbft/gpbft.go	89.12% <81.25%> (-0.15%)	⬇️
gpbft/chain.go	87.87% <85.36%> (-0.55%)	⬇️

... and 2 files with indirect coverage changes

[masih]

Thanks for this PR; the keys look much more readable.

I don't believe this would introduce breaking changes anywhere. I'm specifically thinking about the observer data already collected.

@Kubuxu can you think of any?

[Kubuxu]

I would only be worried about end-users, but given that things are not active yet, the impact would be negligible.

[Stebalien]

I'd definitely do this. I'd also:

1. Omit the power table deltas when empty, or set them to an empty array. Anything but null.
2. Consider changing commitments to be multibase-encoded.

[rvagg]

OK, a few edits and some more additions to get the fully desired output from Lotus. Here's what the current output is for Filecoin.F3GetLatestCertificate. Please have a careful look over this:

{
  "GPBFTInstance": 43253,
  "ECChain": [
    {
      "Key": [
        { "/": "bafy2bzacedjge4udf4ti5uqqhclqj4nhoolcoeonrysoy226hbdww4fhg2rkk" },
        { "/": "bafy2bzacedv47mijo2ipmsnf3fpqpvoyfjmtfrwp67356kmivjfyugfkwlo2g" },
        { "/": "bafy2bzaceciho5erd6n6mapqwagpa5u34zylf6kdtiajlq2lbjzxugn6c5fse" }
      ],
      "Commitments": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
      "Epoch": 2176643,
      "PowerTable": { "/": "bafy2bzaceb4gykw257fov6df5mnyjsvpjxkpapzyxmguexwcmwzf72pued5cw" }
    }
  ],
  "SupplementalData": {
    "Commitments": "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
    "PowerTable": {  "/": "bafy2bzacebf32gcvjz7qy65yz634ekyng5hknegnytda7z6wfm5seoaqebbj4" }
  },
  "Signers": [ 0, 2, 1, 2, 1, 1 ],
  "Signature": "qrzTiqqfeK5ywyt1nY0/7m6/HAEUykw3AYUuTvlsb7nRIZnTX7DueIO3XyKRd4IME0DzLPTnBUkOQ9qA+d//n++NV6rjuznMCHN32ub8DZTC3yVuqeZGWOVAbstBrHbT",
  "PowerTableDelta": [
    {
      "ParticipantID": 138097,
      "PowerDelta": "-839889444667392",
      "SigningKey": null
    }
  ]
}

[rvagg]

Commitment is standard Go base64-padded encoded btw, not multibase. I'm not sure it makes much sense to multibase it unless it's going to be typed as a multibase in Go too. Which would be fine by me but I think this is baked into the protocol now as 32 plain bytes?

[Kubuxu]

It is baked as 32 bytes explicitly.

[Stebalien]

Commitment is standard Go base64-padded encoded btw, not multibase. I'm not sure it makes much sense to multibase it unless it's going to be typed as a multibase in Go too. Which would be fine by me but I think this is baked into the protocol now as 32 plain bytes?

It's baked into the protocol as 32 bytes, but the protocol doesn't really care about the json representation. But leaving it as-is is also fine.

[rvagg]

@masih would you mind having another look now that I've expanded scope please?