Skip to content

学习了下persistent table相关实现,但发现没有compaction机制,一直顺序追加写磁盘空间不会爆嘛 #9272

Answered by liujuncheng
demonatic asked this question in Q&A
Discussion options

You must be logged in to vote

看persistent table实现主要是把要写入的value组织成block顺序追加写入value文件,同时把key和value的序号插入hashmap中用作查询,但没有看到有类似compaction的机制来把key的旧value删除掉,这样一直写磁盘空间不会爆吗,望指点

非常感谢您的反馈,我们后续计划引入 compaction 机制,目前我们提供了 persistent table
读写 api, https://oneflow.readthedocs.io/en/master/one_embedding.html#persistent-read-write ,可以训练结束后,将某个 snapshot 通过读写到新的位置实现 compaction 的功能。

Replies: 1 comment

Comment options

You must be logged in to vote
0 replies
Answer selected by liujuncheng
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Category
Q&A
Labels
None yet
2 participants