Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: networked-dot-storage #18

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Conversation

hannahhoward
Copy link
Member

defining what decentralization means for W3S
Copy link
Contributor

@vasco-santos vasco-santos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hannahhoward thanks for putting this together!

rfc/networked-dot-storage.md Outdated Show resolved Hide resolved

### No Data Loss

1. Users should be able to specify how many unique copies of their data they want backed up, and the W3S network should handle insuring that many copies remain backed up as long as the user pay to store in W3S.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some food for thought: more than replication, I have always liked the idea of allowing user custom region, or even selecting providers with different SLAs. For instance, one node running this service could host things in S3 EU Region, with greater SLA characteristics. Storage costs SHOULD not be tied to what web3.storage says, but what each provider offers to enable a more open market

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 - I'd even add in - in addition to storing on N Filecoin Storage Providers, I'd like to be able to back my data up to S3, R2 or other S3-api providers for a fee - I think having a backup that is not vulnerable to a catastrophic failure of the Filecoin network and instead relies on the social-proof-resiliance of keeping a copy on a major internet service provider is something that users (at least me!) would pay for!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a nuance, but I can't help but wonder if making users choose number of unique copies is the right approach, my guess is users want some resilience, perhaps geographic distribution and other forms of assurances that data availability is adequate. Trying to figure out number of replicas seems several steps disconnected from those needs and also would remove some flexibility on the implementer side.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah honestly I think this should be a much more nuanced thing - imho different w3up providers should be able to offer different levels of "quality of service" on a bunch of different axes (latency, resilience, bandwidth, etc) as well different levels of legal/social guarantees (say, "we will comply with GDPR" or "we will never host data in Iran/China/Russia/USA/etc") - we could then craft different "default" user experiences for people that connected them with providers that met their needs, and once they are in the ecosystem they wouldn't necessarily need to maintain a relationship with us if they don't want to - I think this could go a long way toward balancing "we want to build a defensible business" and "we want to minimize our role as a central point of failure or point of centralization for our users" which imho are in dramatic tension for us...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh also I think this might even open up space for people to make decisions about storage providers based on real-world social factors, which is imho what is needed to support "private data" or PII - this is honestly the market segment I am most excited about

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a nuance, but I can't help but wonder if making users choose number of unique copies is the right approach, my guess is users want some resilience, perhaps geographic distribution and other forms of assurances that data availability is adequate. Trying to figure out number of replicas seems several steps disconnected from those needs and also would remove some flexibility on the implementer side.

Yes, +1. In the same way that I do not care how many replicas AWS stores of the data I have in S3. It should be adequete, so that the data is not lost.

When we talk about geographical distribution, my suspicion is that we're mostly thinking about retrieving the data quickly, which is what the CDN side of the product should be oriented around, not the storage side.

rfc/networked-dot-storage.md Outdated Show resolved Hide resolved

One proposed path forward is to not pursue "decentralization" beyond trying to make other nodes a reality, perhaps through incentivization, and leaning heavily into product improvements that differentiate our platform. We could aim for "S3 if AWS invested in open source S3 implementations and made it content addressed".

This is not the path we are going to pursue. It feels important to say that up front.
Copy link
Member

@travis travis Mar 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not the path we are going to pursue.

Well, it's funny - what's described below is a lot closer to what I meant by "S3 if AWS invested in open source S3 implementations and made it content addressed" than what I understood we were aiming to build, and the differences are things I feel pretty positive about - coordinating different w3up operators in order to make the user experience really good, exposing more of the details of what Filecoin is providing (eg, teaching the end user about the proofs that their data is replicated across the Filecoin network and exposing them in a user friendly way in our UI), data portability and low barriers to entry to getting a node running, etc [ed: after re-reading I'm realizing that even some of the things I've listed feel in scope of "S3 if AWS invested in open source and built it on content addressing", so, even better! ]

tbh after reading this line I was expecting to be disappointed by the rest of this doc but it's just the opposite - maybe just a masterful use of expectations ;-p

anyway TLDR I'm pretty positive about this direction, thanks for putting this together! will leave more thoughts inline below...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fair.

I will say something, which I think many people are talking about similar underlying things through different language, leading to a perception of difference in goals that is outsized (even if there are real, if smaller differences underneath). My thought is that describing user goals is what can produce better conversations.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

people are talking about similar underlying things through different language, leading to a perception of difference in goals that is outsized

agree 100% - super grateful for this document as I think it does a great job of

describing user goals is what can produce better conversations.


### Perpetuity

1. Users should be able to store hot data in W3S that will be available for retrieval until they stop paying for it. They should not have to worry about any single business entity holding their data disappearing. If it does, the data should go somewhere else without their needing to intervene.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love this. At some point I'd really like to game out "what would it look like if w3s shut down suddenly" from the perspective of our users. Really love the idea of centering "our users won't be screwed if we go away" and I'm sensitive to the fact that this is in tension with the traditional startup virtue of having a "defensible business" - I think there is going to be pressure from investors to ensure that our users are dependent on us and can't leave and I think we should be ready to push back on that when it comes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If w3s goes away there is no longer an entity user was paying for service, which would mean user stopped paying so data can disappear, is it not ?

I also wonder if entity providing this assurance should be different from the w3s itself ? In my imagination that entity will be cooperating with w3s and other instances of the protocol deployment to ensure that data can be migrated in case one of them is out of business.

Another question that comes in my head is how can data be kept available in case of entity going out of business as there may not be other entity that offers service for the same price.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If w3s goes away there is no longer an entity user was paying for service, which would mean user stopped paying so data can disappear, is it not ?

Well right so I think it's critical that we figure out how we can let users pay for storage on the "w3s network" without being dependent on w3s-the-legal-entity - this doesn't have to be the default experience, but for users who would like to be resilient to our entity failing I think this would be a critical requirement. IMHO we should try to go out of our way to enable this as I think it's potentially a key differentiator for our product compared to traditional storage solutions and competitors.

Copy link
Member Author

@hannahhoward hannahhoward Apr 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If only there were some sort of mechanism for a public distributed ledger of monetary transactions whose governance and existence was not entirely dependent on the success or failure of a single business entity, we could maybe use that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Users should be able to store hot data in W3S that will be available for retrieval until they stop paying for it.

Hmm. I don't think we should be replicating what filecoin does, but that is perhaps not what is meant here?

I think we should be saying that users can pay to ensure their data is hot-retrievable. I think that positions us NOT as a competitor. Hot retrievability is not something that filecoin guarantees at a network level...but is what folks actually want.

As such I'm more pro building an ingestion network (aka temporary-storage-on-the-way-to-filecoin) and the hot retrieval network (that can read from ingestion nodes_or_ filecoin SPs). The former feels like a much simplier and achievable prospect than storing data perpetually, that does not conflict with what filecoin is already doing. The latter is the real opportunity we have to enable instant and continuous hot retrievability from "filecoin" (in quotes here because it would be from a combintation of our network AND filecoin).


### Sensible defaults with customization

1. Users should have the option to store with simply "the W3S network" rather than a specific business entity or Filecoin SP. This is a hard lesson learned from Filecoin that forcing users to choose from a marketplace where information on available options is hard to come by is a terrible user experience.
Copy link
Member

@travis travis Mar 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Users should have the option to store with simply "the W3S network" rather than a specific business entity or Filecoin SP

ok you've got me! 🚫 ⛓️ 💊

forcing users to choose from a marketplace where information on available options is hard to come by is a terrible user experience.

💯 💯 💯

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree 100%


## But who wants this?

A common refrain in response to some of the positive properties that "decentralization" might enable is "none of our existing users asked for this".
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Speaking for myself, the primary thing that I don't think anyone is asking for is "I want the bits of my upload to be stored in hot storage hosted by random participants in the w3s network."

I'm very sympathetic to the idea that people don't always want to think about where their bits actually live, so I do think we should provide a "default" storage location and I don't necessarily think the healthiest way to build that is to have us (eg, w3s-maintained S3 or equivalent) be the one and only default, BUT I do think that things get a lot lot lot more complicated and risky when we try to rely on an incentivized permissionless network for this, imho in a way that is out of proportion to the amount of apparent market demand for this specific property.

Data portability, verifiability of actions taken on the network, sensible defaults with the ability to customize, storage perpetuity - all of these things seem (to me) clearly valuable and desired by our customers and the market at large, but I don't think any of them require a part of our system that can distribute the actual bits to a random group of w3up operators. The bits that might utilize this - a desire to avoid becoming a monopoly (defacto or otherwise), resiliance in the face of a catastrophic failure of the w3s organization - do seem valuable to me, but I'm not convinced that "a permissionless network of hot storage nodes as the default destination for the bits" is the simplest or most elegant solution to either of these. Maybe I'm wrong - maybe this is the only way to achieve these goals! - but I would definitely like to spend more time kicking this around before we go down this road, because it feels like a very very risky path to take.

Copy link
Member Author

@hannahhoward hannahhoward Mar 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eventually, I see a heterogenous network that employs a mix of permissionless nodes as well as more trusted gaurantors of service quality. At minimum, if a user gives us data, we shouldn't hand it out randomly -- we should hand it to those who have the best reputation.

I've noticed there's a certain assumption here of "we'll build it like Saturn" which is why I wanted to specifically address that below. I absolutely agree it's silly to scale to a huge permissionless network until there is demand. I call out the lack of economic connection between supply and demand to be perhaps Saturn's biggest mistake. At the same time, I don't personally think an entirely federated system is the end state. Does it look like Saturn? Probably not. But I do want us to get to a world where anyone can be part of it, and at least take a shot at offering a better service for a lower price, whether they are large or small.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just want call out that that there is an implicit assumption that w3s is network that various providers can join and participate. Irony is that picture is very centralized around w3s entity. What I would like to see is multiple networks like w3s that can co-operate and providers that can be part of more then one such network.

Does multiple w3s instances sounds more like federated network, but you could also see it as distributed network as service providers that are connected to various w3s like hubs that act as a distributors.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what I'd like to see is:

  1. we set up this network and serve as the default point of entry for people
  2. we continually move to make it possible for people to decouple themselves from w3s-the-legal-entity and make this a core value proposition of our service - we will constantly need to balance this against the needs and desires of stakeholders in w3s-the-legal-entity so we'll need to be thoughtful about how we maintain this line
  3. in the long run steady state, w3s-the-legal-entity strives to continually improve the "default experience" for engaging with w3s-the-network while simultaneously encouraging an ecosystem of businesses to grow around w3s-the-network - some of these businesses may be addressing market segments that we choose not to engage in, while others may directly compete with us - this is a feature and something we will ideally make a core value proposition of w3s-the-network

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: 3

Yes, this how I think of it.

For me the end state of the w3s entity is some kind of foundation where a small part of protocol revenue pays a core team to keep developing the network.

The nice thing is once the token is liquid, if we end up with a token and a token raise, the VCs have already gotten their exit. This to me is the moment you can transition to a model that more closely resembles https://github.com/web3-storage/RFC.staging/pull/1 . This is all of course fantasy, based on everything going exactly how it's supposed to, not getting caught up in some kind of crypto bubble, but that actually is how I would like it to go.


### Distribution of resources

6. Users should not have to worry about large service providers establishing monopolies that reduce user choice or drive up costs. To insure this, W3S should emphasize data portabiltity, fairness in data distribution, and low barrier to entry for new providers.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

data portabiltity

Love.

low barrier to entry for new providers

Love!

fairness in data distribution

This is the part I don't really get. Fairness in what sense?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Essentially, for a user who doesn't want to make choices about their data, they should have assurance the network isn't betting only on large players, and is insuring they get the best service regardless of size. And, they should be able to verify that.


*** But isn't this a point of centralization if everyone uses it? ***

Absolutely. It's important that key coordination functions in the network like router.storage eventually operate through distributed consensus and produce verifiable results.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

operate through distributed consensus

In this future, isn't the consensus mechanism itself then a point of centralization and a single point of failure? If the blockchain that does this consensus is down, does that mean the network stops working? A storage network like this feels like something that should be able to splinter and fragment to adapt to the needs of sub-groups, and a single (even permissionless and distributed) consensus mechanism feels to me like it sits in tension with some of the user-facing benefits of "decentralization." Maybe IPC solves this!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally I do think some of the large L1s can be considered pretty reliable at this point.

Filecoin hasn't halted since 2020
Ethereum went a year without halting (surprising though it had one this recently)
Bitcoin's last halt was apparently long enough ago that I can't find any information on it.

I will assure everyone we will not be building shit on Solana which breaks all the time. :P

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: other stuff -- again, I think there's a complicated issue around whether people want choice or simply something that distributes things fairly. I just disagree there isn't value to consensus in this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

disagree there isn't value to consensus in this

totally on the same page - I probably come off as a blockchain hater but inside of me is a computer scientist that thinks they are the coolest thing and desperately wants to find really cool ways to use them. I've just found that it's safer to be a blockchain skeptic by default and to try solving problems nearly any other way before committing to this route. I genuinely hold a lot of hope that we have a real usecase for them here!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a the hot take: Job of the router is to uphold user preferences and load balance supply and demand. Assuming that is accurate do we even need blockchain, could that simply be an IPVM task handling routing ? That way users gain a choice to pick the "router" that they find to be a right fit. In fact user with right technical skill could even fork the router to create one that fits their needs if one does not exist already.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I'm open to ideas here. The first task is just to build a router of any kind. If it will run in easily in a WASM environment, bonus, cause then there is a path to either IPC OR IPVM, once the right path becomes clear.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if someone wants to write the router in rust, I'm open to it. As long as it doesn't use rust async, which I think won't work well in either env.


*** But isn't this a point of centralization if everyone uses it? ***

Absolutely. It's important that key coordination functions in the network like a repair service eventually operate through distributed consensus and produce verifiable results.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar question as above re: distributed consensus - I don't think this is a point of centralization if there are other nodes on the network that can perform this service for people. Maybe the blockchain provides a coordination point so people can easily hand off to other "repair services" - some level of that makes sense to me, though I still worry about it being a single point of failure in that process!

Copy link
Member Author

@hannahhoward hannahhoward Mar 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea perhaps repair service isn't the right thing for a blockchain, cause you generally might have some level of choice there.

here's the way I look at it:

  • single service = single point of failure
  • multiple services = no single point of failure, lots of choices, but no coordination, which often reduces user value
  • blockchain = acts like single service, but is actually resilient no single point of failure system, and produces more trustworthy output (usually) -- there are times you want this, times you really just want to give people choices.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about alternative view of the repair service. Instead of having a single blockchain operated system, allow users to run repair service anywhere they like, IPVM, AWS, on user device in their closet or whatever else (in fact I think we could have interesting offering here). In a sense repair service become an agent hired by user which acts on user behalf.

Point I'm trying to make here is that coordination is both difficult and costly and I'm not sure it provides any real benefit to the user that needing assurance of service been operational. Also note that if repair service is just a code you can run it can probably also run as smart contract if users would want to do that.

Overall I'm for plurality of choice, as opposed to assumption that there is a universal solution that fits everyones needs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I get that choice is your go to.

Personally, I feel that the vast majority of every day users are not going to want to get down and dirty on these details, which I think are pretty low level. Which is why I lean towards sensible defaults first. But I'm not opposed to choice.

To be clear though I think that:

  1. Repair service is absolutely something there can be many of.
  2. I really misspoke, the default repair service SHOULD simply be a smart contract.


*** But isn't this a point of centralization if everyone uses it? ***

Absolutely. It's important that key coordination functions in the network like w3-filecoin eventually operate through distributed consensus and produce verifiable results.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto, same, "I used a blockchain to get your data to the blockchain" decentralized inception?!

but tbh of all the "exit to consensus" questions this one makes the most sense to me? the filecoin pipeline is already so async that a blockchain outage slowing it down feels fine, and having all of the intermediary steps show up on chain somewhere seems pretty reasonable.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean this service would likely run as a Filecoin L2, so really it's just a sub routine :P


### Is decentralization the wrong word to use?

Decentralization is both a concept so overloaded it lacks any coherence, and a narrative that our product team has validated is indeed important to many market segments in web3. Actual customers have told us they won't use our product unless it's "decentralized". Likely, in some cases this is a novel kind of CYA (cover-your-ass) -- where enterprise customers want SOC-2 compliance in a data center so they can blame someone else if data gets lost, web3 customers want services they use to be "decentralized" so they can't get called out by the internet for not being "web3". Decentralization is both an annoying meme and an important narrative we have to contend with.
Copy link
Member

@travis travis Mar 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actual customers have told us they won't use our product unless it's "decentralized".

Right totally, it's just that as you describe they all mean slightly different things, so when they say this we need to dig deeper and figure out which bits they genuinely find important. Our product team coming back from a call with a customer and saying "they won't use us unless we're decentralized" is worse than no feedback at all - it just throws a log on every little fire we each maintain to our preferred definitions of the word and makes these conversations harder and less concrete. I don't think we necessarily need to stop using the word entirely, but the phrase "we need to do X because it makes us more decentralized and that's what our customers want!" is up there with "electrolytes, it's what plants crave!" in the pantheon of misguided and misleading sentiments.

Copy link
Member Author

@hannahhoward hannahhoward Mar 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may be missing part of the point of the last sentence of this paragraph. "decentralization" as narrative is important whether or not it has a clear meaning. Concepts and marketing terms can at once be silly and simultaneously be important -- i.e. something you shouldn't ignore -- if you want to be successful.

What I am attempting to do is define internally what we view as important to decentralization. We will still market it outwardly as "decentralized" and as we do it attempt to shape the narrative of what decentralization means. As I said in the "who wants this section", many people have trouble knowing concretely what is possible. I'd like to believe that in addition to just CYA, at least part of the enthusiasm for decentralization is simply an ill defined impulse for a better kind of service on the internet. Maybe I'm too much of an optimist.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair, and fwiw I don't disagree with any of this - I am mostly just harping on internal use of "decentralization" without more context on what we mean specifically when we're talking about the nuts and bolts of what we want to build and why - totally agree that we can and should be taking advantage of its value as a marketing meme!

If anything I think we have been too shy about highlighting the parts of the current state of the project that imho are "decentralized" in meaningful ways simply because our backend happens to be implemented in terms of a well known cloud service provider rather than being "hosted on the decentralized web" somehow - open specs, open software, (nearly) trivial deployment of third party w3up instances are all things we have right now that address what I think are the interesting user-facing facets of "decentralization" and I think we should lean into that.

part of the enthusiasm for decentralization is simply an ill defined impulse for a better kind of service on the internet

hard agree, this is why I'm here for sure! and the reality of this is part of why I get so worked up by imprecise usage of this term when we talk about what we want to build - I think there's a lot of real value here that tends to get overshadowed by the marketing/vibes-y usages of it, and I really want to be building things that address this ill-defined desire rather than something that just uses the term to get pageviews - I think I'm in good company on that front!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just declare ourselves decentralized ? That way customers can tell us what they actually need or tell us how what we are doing is not decentralized enough which is effectively telling us what they need.


Decentralization is both a concept so overloaded it lacks any coherence, and a narrative that our product team has validated is indeed important to many market segments in web3. Actual customers have told us they won't use our product unless it's "decentralized". Likely, in some cases this is a novel kind of CYA (cover-your-ass) -- where enterprise customers want SOC-2 compliance in a data center so they can blame someone else if data gets lost, web3 customers want services they use to be "decentralized" so they can't get called out by the internet for not being "web3". Decentralization is both an annoying meme and an important narrative we have to contend with.

I would say what we're really building is an incentivized permissionless cooperative distributed network for content addressed data. If anyone wants to try to market that, I encourage them to. But internally, maybe we should consider not saying decentralization as much. At minimum, let's align on what we're trying to achieve for our users.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

incentivized permissionless cooperative distributed network for content addressed data

IPCDNCAD.... it's got IP, CDN and CAD in it 🤣

it is definitely wordy, but honestly I think this is a great starting point for thinking about how we do market this - different pieces will matter to different people but hey - more markets we can address!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the idea of trying to market that phrase was intended as a joke :P

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but please, have a go.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hahaha - it's probably a little too wordy to be a single marketing meme, but we should probably break it down and see what kinds of fun memetics we can get out of combinations of it - a cooperative network for content addressed data sounds dope as heck!


1. Users should have the option to store with simply "the W3S network" rather than a specific business entity or Filecoin SP. This is a hard lesson learned from Filecoin that forcing users to choose from a marketplace where information on available options is hard to come by is a terrible user experience.

2. Users should still have the option to choose a provider they like, or to move between providers.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean filecoin storage provider here or something else ? I think interesting question in my head is do users make deals with w3s and w3s makes deals with SPs or do we want to allow users to make deals with SPs directly ? I recall requests for later on few occasions although I am not sure what the use cases were.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think users who just want to store with an SP directly should have that option.

I've come around to the fractal design concept -- an SP can run Filecoin.storage (or whatever we call Filecoin SP tailored w3up). If they do this, and they want to find their own users, they could just tell them to make direct data uploads to their instance of w3up, which would go in their own aggregation pipeline, and then eventually get put in their own deals for their Filecoin SP operation. and/or, they could connect their filecoin.storage instance to a web3.storage router, and get a portion of uploads going through us, which would likely just get aggregated along with any direct uploads they received.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But seperately, you can also work directly with a provider running a hot storage instance with no Filecoin operation of their, who would then use a common aggregation pipeline (likely the one we run for a while).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is in conflict with "store with the W3S network". IMO if folks want to use a specific provider they should do that - it is not our offering. Or alternatively like you say, setup their own network using our OSS that only uses specific providers.


4. A user paying for multiple backups should be able to verify each backup is still functioning.

5. If verification demonstrates that a particular provider is failing to provide a service that the user has paid for, the network should detect the failures and move their data automatically. (or alternatively, the user should have the ability to create a dispute and move to a different provider while getting money back)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be and not or, if someone failed to uphold commitment user should be able to hold them accountable. In other words repair should take place but user still should be able to request compensation for service disruption if they so choose.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for sure.


### Does "distributed consensus and produce verifiable results" mean smart contracts on a blockchain?

Most likely, yes. But I'm open to hearing about other consensus mechanisms. That said, for the specific problem of trustless consensus on limited execution of relatively small rule sets, honestly, I think blockchains are the most proven solution to date. We need to assess objectively the risks/benefits of a different approach.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it might be worth teasing out what do we need consensus on ? Perhaps some things do not need consensus but could instead lean into giving user a choice ? I think it is not a stretch to describe blockchains as extremely inefficient coordination system with some mutable state. Which makes me wonder if by consensus we mean coordination and if so perhaps what we actually need is to expose (mutable) state with some concurrency controls.

Copy link
Contributor

@Gozala Gozala left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for describing coherent picture & distilling concrete goals. I do like it, that said still think there is few misalignments which I called out in the comments and will capture in the short list below:

  1. I still think one network is not a right direction and we should be thinking about plurality of networks that may overlap. I suspect multiple networks will arise regardless, however designing for that future may facilitate cooperation and interop as opposed to just competition.
  2. I think it is worth considering and exploring whether we really need network level consensus. I would argue that reducing amount of consensus needed will greatly improve network resilience and adaptability. I also make a point that perhaps coordination should not be function of consensus, but rather a function of user preference.

@hannahhoward
Copy link
Member Author

hannahhoward commented Apr 4, 2024

  1. I still think one network is not a right direction and we should be thinking about plurality of networks that may overlap. I suspect multiple networks will arise regardless, however designing for that future may facilitate cooperation and interop as opposed to just competition.

I am not opposed to multiple networks. Everything should always remain open. I am fairly strongly in favor of building one network first. I don't believe multiple networks will emerge until a more mature state of the network.

  1. I think it is worth considering and exploring whether we really need network level consensus. I would argue that reducing amount of consensus needed will greatly improve network resilience and adaptability. I also make a point that perhaps coordination should not be function of consensus, but rather a function of user preference.

I think we need network level consensus where money is concerned. If a user pays the network to store some data, and nodes are going to get paid for holding that data, there needs to be a mechanism to be able to audit that the data was handed out fairly by a router, and payments flowed where they were supposed to go, according the rules of compensation we set up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants