学习了下persistent table相关实现,但发现没有compaction机制,一直顺序追加写磁盘空间不会爆嘛 #9272
-
看persistent table实现主要是把要写入的value组织成block顺序追加写入value文件,同时把key和value的序号插入hashmap中用作查询,但没有看到有类似compaction的机制来把key的旧value删除掉,这样一直写磁盘空间不会爆吗,望指点 |
Beta Was this translation helpful? Give feedback.
Answered by
liujuncheng
Oct 19, 2022
Replies: 1 comment
-
非常感谢您的反馈,我们后续计划引入 compaction 机制,目前我们提供了 persistent table |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
liujuncheng
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
非常感谢您的反馈,我们后续计划引入 compaction 机制,目前我们提供了 persistent table
读写 api, https://oneflow.readthedocs.io/en/master/one_embedding.html#persistent-read-write ,可以训练结束后,将某个 snapshot 通过读写到新的位置实现 compaction 的功能。