[fixbug]: Fixed the issue in MinhashBuildIndex where get_datafolder w…

…as not used to obtain DataFolder for input_folder and output_folder. (#307) Fixed the issue in MinhashBuildIndex where get_datafolder was not used to obtain DataFolder for input_folder and output_folder.
huggingface · Nov 27, 2024 · fe81883 · fe81883
1 parent 371c014
commit fe81883
Showing 1 changed file with 2 additions and 2 deletions.
diff --git a/src/datatrove/pipeline/dedup/minhash.py b/src/datatrove/pipeline/dedup/minhash.py
@@ -574,8 +574,8 @@ def __init__(
         lines_to_buffer: int = 5,
     ):
         super().__init__()
-        self.input_folder = input_folder
-        self.output_folder = output_folder
+        self.input_folder = get_datafolder(input_folder)
+        self.output_folder = get_datafolder(output_folder)
         self.config = config or MinhashConfig()
         self.index_name = index_name
         self.lines_to_buffer = lines_to_buffer