Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a homeDir field in LocalFileSystem for removefile to check #4543

Open
wants to merge 35 commits into
base: master
Choose a base branch
from

Conversation

SterlingT3485
Copy link
Collaborator

@SterlingT3485 SterlingT3485 commented Nov 18, 2024

Description

Added this for removeFileIfExists(const std::string& path) to only remove the files under our DBpath

Fixes #4529

Contributor agreement

Copy link

Benchmark Result

Master commit hash: 6f1bf5b60107419eb2f4d947ca0fd8ed477f72b0
Branch commit hash: d883d92364e6b10602b5368c24e50aa6a3e32a83

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 650.13 642.95 7.18 (1.12%)
aggregation q28 11646.45 11469.90 176.55 (1.54%)
filter q14 132.24 126.43 5.81 (4.60%)
filter q15 148.51 123.37 25.14 (20.38%)
filter q16 313.53 303.19 10.34 (3.41%)
filter q17 454.74 444.82 9.92 (2.23%)
filter q18 1939.53 1946.70 -7.17 (-0.37%)
filter zonemap-node 94.53 86.57 7.96 (9.19%)
filter zonemap-node-lhs-cast 94.74 86.53 8.21 (9.49%)
filter zonemap-rel 5518.68 5743.59 -224.91 (-3.92%)
fixed_size_expr_evaluator q07 553.95 544.45 9.50 (1.74%)
fixed_size_expr_evaluator q08 772.95 778.80 -5.85 (-0.75%)
fixed_size_expr_evaluator q09 770.99 774.16 -3.18 (-0.41%)
fixed_size_expr_evaluator q10 249.04 238.02 11.02 (4.63%)
fixed_size_expr_evaluator q11 244.31 237.01 7.30 (3.08%)
fixed_size_expr_evaluator q12 241.49 236.82 4.68 (1.98%)
fixed_size_expr_evaluator q13 1484.91 1503.38 -18.47 (-1.23%)
fixed_size_seq_scan q23 122.98 119.91 3.06 (2.56%)
join q29 668.89 662.44 6.44 (0.97%)
join q30 1448.48 1289.55 158.93 (12.32%)
join q31 8.07 7.01 1.06 (15.19%)
ldbc_snb_ic q35 475.74 406.73 69.01 (16.97%)
ldbc_snb_ic q36 129.91 126.12 3.79 (3.01%)
ldbc_snb_is q32 5.71 5.16 0.54 (10.56%)
ldbc_snb_is q33 13.51 15.12 -1.60 (-10.60%)
ldbc_snb_is q34 1.23 1.61 -0.38 (-23.88%)
multi-rel multi-rel-large-scan 1755.35 1612.54 142.81 (8.86%)
multi-rel multi-rel-lookup 9.90 21.53 -11.62 (-53.99%)
multi-rel multi-rel-small-scan 96.10 99.58 -3.48 (-3.49%)
order_by q25 151.85 131.52 20.33 (15.45%)
order_by q26 476.89 460.60 16.29 (3.54%)
order_by q27 1439.51 1435.41 4.10 (0.29%)
scan_after_filter q01 177.25 174.04 3.21 (1.85%)
scan_after_filter q02 162.96 157.98 4.98 (3.15%)
shortest_path_ldbc100 q37 85.03 84.91 0.11 (0.14%)
shortest_path_ldbc100 q38 462.56 418.99 43.57 (10.40%)
shortest_path_ldbc100 q39 59.51 63.00 -3.49 (-5.53%)
shortest_path_ldbc100 q40 518.75 514.13 4.63 (0.90%)
var_size_expr_evaluator q03 2078.36 2102.75 -24.39 (-1.16%)
var_size_expr_evaluator q04 2258.61 2268.57 -9.96 (-0.44%)
var_size_expr_evaluator q05 2627.50 2699.13 -71.63 (-2.65%)
var_size_expr_evaluator q06 1338.97 1345.60 -6.63 (-0.49%)
var_size_seq_scan q19 1473.81 1479.78 -5.97 (-0.40%)
var_size_seq_scan q20 2575.97 2767.61 -191.63 (-6.92%)
var_size_seq_scan q21 2297.65 2317.91 -20.26 (-0.87%)
var_size_seq_scan q22 127.99 130.39 -2.39 (-1.84%)

Copy link

Benchmark Result

Master commit hash: 6f1bf5b60107419eb2f4d947ca0fd8ed477f72b0
Branch commit hash: d5540757b2ffa68c06a96e450c0c99a5bec34e66

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 651.40 642.95 8.45 (1.31%)
aggregation q28 13747.32 11469.90 2277.42 (19.86%)
filter q14 133.79 126.43 7.37 (5.83%)
filter q15 133.23 123.37 9.86 (7.99%)
filter q16 312.34 303.19 9.15 (3.02%)
filter q17 451.34 444.82 6.52 (1.47%)
filter q18 1964.08 1946.70 17.38 (0.89%)
filter zonemap-node 95.20 86.57 8.63 (9.97%)
filter zonemap-node-lhs-cast 95.33 86.53 8.80 (10.17%)
filter zonemap-rel 5449.94 5743.59 -293.65 (-5.11%)
fixed_size_expr_evaluator q07 553.02 544.45 8.56 (1.57%)
fixed_size_expr_evaluator q08 767.76 778.80 -11.04 (-1.42%)
fixed_size_expr_evaluator q09 765.07 774.16 -9.10 (-1.17%)
fixed_size_expr_evaluator q10 246.02 238.02 8.00 (3.36%)
fixed_size_expr_evaluator q11 241.28 237.01 4.27 (1.80%)
fixed_size_expr_evaluator q12 239.32 236.82 2.50 (1.06%)
fixed_size_expr_evaluator q13 1489.42 1503.38 -13.96 (-0.93%)
fixed_size_seq_scan q23 124.93 119.91 5.02 (4.19%)
join q29 603.19 662.44 -59.25 (-8.94%)
join q30 1356.35 1289.55 66.80 (5.18%)
join q31 6.10 7.01 -0.91 (-12.93%)
ldbc_snb_ic q35 473.78 406.73 67.05 (16.48%)
ldbc_snb_ic q36 132.44 126.12 6.32 (5.01%)
ldbc_snb_is q32 5.29 5.16 0.13 (2.54%)
ldbc_snb_is q33 14.12 15.12 -1.00 (-6.59%)
ldbc_snb_is q34 1.51 1.61 -0.10 (-6.42%)
multi-rel multi-rel-large-scan 1711.76 1612.54 99.22 (6.15%)
multi-rel multi-rel-lookup 39.03 21.53 17.50 (81.32%)
multi-rel multi-rel-small-scan 86.04 99.58 -13.53 (-13.59%)
order_by q25 139.69 131.52 8.17 (6.21%)
order_by q26 458.55 460.60 -2.05 (-0.45%)
order_by q27 1404.78 1435.41 -30.62 (-2.13%)
scan_after_filter q01 174.56 174.04 0.52 (0.30%)
scan_after_filter q02 163.82 157.98 5.83 (3.69%)
shortest_path_ldbc100 q37 100.30 84.91 15.39 (18.12%)
shortest_path_ldbc100 q38 504.63 418.99 85.64 (20.44%)
shortest_path_ldbc100 q39 59.56 63.00 -3.44 (-5.45%)
shortest_path_ldbc100 q40 547.42 514.13 33.30 (6.48%)
var_size_expr_evaluator q03 2071.98 2102.75 -30.76 (-1.46%)
var_size_expr_evaluator q04 2237.72 2268.57 -30.85 (-1.36%)
var_size_expr_evaluator q05 2616.84 2699.13 -82.29 (-3.05%)
var_size_expr_evaluator q06 1334.99 1345.60 -10.61 (-0.79%)
var_size_seq_scan q19 1458.08 1479.78 -21.70 (-1.47%)
var_size_seq_scan q20 2503.87 2767.61 -263.74 (-9.53%)
var_size_seq_scan q21 2282.35 2317.91 -35.55 (-1.53%)
var_size_seq_scan q22 126.76 130.39 -3.62 (-2.78%)

Copy link

Benchmark Result

Master commit hash: 6f1bf5b60107419eb2f4d947ca0fd8ed477f72b0
Branch commit hash: 9d47d967cdf51a9b1f4a8df3a07e0c1c025ffacd

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 643.63 642.95 0.68 (0.11%)
aggregation q28 11796.71 11469.90 326.80 (2.85%)
filter q14 127.74 126.43 1.32 (1.04%)
filter q15 133.91 123.37 10.53 (8.54%)
filter q16 299.44 303.19 -3.76 (-1.24%)
filter q17 444.70 444.82 -0.12 (-0.03%)
filter q18 1927.93 1946.70 -18.78 (-0.96%)
filter zonemap-node 86.18 86.57 -0.39 (-0.45%)
filter zonemap-node-lhs-cast 86.41 86.53 -0.12 (-0.14%)
filter zonemap-rel 5541.93 5743.59 -201.66 (-3.51%)
fixed_size_expr_evaluator q07 545.64 544.45 1.19 (0.22%)
fixed_size_expr_evaluator q08 760.51 778.80 -18.28 (-2.35%)
fixed_size_expr_evaluator q09 760.26 774.16 -13.91 (-1.80%)
fixed_size_expr_evaluator q10 240.89 238.02 2.87 (1.21%)
fixed_size_expr_evaluator q11 235.89 237.01 -1.13 (-0.47%)
fixed_size_expr_evaluator q12 235.12 236.82 -1.69 (-0.71%)
fixed_size_expr_evaluator q13 1474.36 1503.38 -29.02 (-1.93%)
fixed_size_seq_scan q23 122.83 119.91 2.92 (2.44%)
join q29 629.12 662.44 -33.32 (-5.03%)
join q30 1370.63 1289.55 81.08 (6.29%)
join q31 7.13 7.01 0.12 (1.74%)
ldbc_snb_ic q35 412.21 406.73 5.48 (1.35%)
ldbc_snb_ic q36 137.77 126.12 11.66 (9.24%)
ldbc_snb_is q32 3.37 5.16 -1.79 (-34.70%)
ldbc_snb_is q33 12.51 15.12 -2.61 (-17.23%)
ldbc_snb_is q34 1.61 1.61 0.00 (0.31%)
multi-rel multi-rel-large-scan 1871.10 1612.54 258.56 (16.03%)
multi-rel multi-rel-lookup 42.23 21.53 20.70 (96.17%)
multi-rel multi-rel-small-scan 87.06 99.58 -12.51 (-12.57%)
order_by q25 138.36 131.52 6.84 (5.20%)
order_by q26 452.70 460.60 -7.91 (-1.72%)
order_by q27 1414.26 1435.41 -21.15 (-1.47%)
scan_after_filter q01 173.72 174.04 -0.33 (-0.19%)
scan_after_filter q02 155.86 157.98 -2.13 (-1.35%)
shortest_path_ldbc100 q37 88.61 84.91 3.70 (4.36%)
shortest_path_ldbc100 q38 498.07 418.99 79.08 (18.87%)
shortest_path_ldbc100 q39 63.87 63.00 0.87 (1.37%)
shortest_path_ldbc100 q40 543.76 514.13 29.63 (5.76%)
var_size_expr_evaluator q03 2034.77 2102.75 -67.98 (-3.23%)
var_size_expr_evaluator q04 2257.84 2268.57 -10.73 (-0.47%)
var_size_expr_evaluator q05 2593.44 2699.13 -105.68 (-3.92%)
var_size_expr_evaluator q06 1320.63 1345.60 -24.97 (-1.86%)
var_size_seq_scan q19 1438.27 1479.78 -41.51 (-2.81%)
var_size_seq_scan q20 2599.84 2767.61 -167.77 (-6.06%)
var_size_seq_scan q21 2289.41 2317.91 -28.49 (-1.23%)
var_size_seq_scan q22 128.18 130.39 -2.21 (-1.69%)

Copy link

codecov bot commented Nov 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.36%. Comparing base (6157e57) to head (1c1bb32).
Report is 4 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4543      +/-   ##
==========================================
+ Coverage   87.34%   87.36%   +0.02%     
==========================================
  Files        1348     1349       +1     
  Lines       56508    56639     +131     
  Branches     7112     7131      +19     
==========================================
+ Hits        49357    49484     +127     
- Misses       6981     6983       +2     
- Partials      170      172       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.


🚨 Try these New Features:

Copy link
Collaborator

@acquamarin acquamarin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also add some tests.

src/main/database.cpp Outdated Show resolved Hide resolved
src/main/database.cpp Show resolved Hide resolved
src/include/common/file_system/local_file_system.h Outdated Show resolved Hide resolved
src/include/common/file_system/local_file_system.h Outdated Show resolved Hide resolved
auto subStr = subPath.generic_string();

// Check if subStr starts with baseStr and ensure proper boundaries
return subStr.starts_with(baseStr) &&
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the string based comparison is safe?
I will let @mewim comment on this

Copy link
Collaborator Author

@SterlingT3485 SterlingT3485 Nov 19, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we already guaranteed the path is a standard form absolute path that contains no .. or . , also checked the length of two path and their ending characters?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to a safer solution , which uses filesystem::relative to check the relation between two absolute dirs, should be safe enough

Copy link

Benchmark Result

Master commit hash: 6f1bf5b60107419eb2f4d947ca0fd8ed477f72b0
Branch commit hash: 4ad40bbdfae5530a06d7a1656179acc4ac234323

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 648.68 642.95 5.73 (0.89%)
aggregation q28 11922.53 11469.90 452.63 (3.95%)
filter q14 135.17 126.43 8.74 (6.91%)
filter q15 135.40 123.37 12.03 (9.75%)
filter q16 306.19 303.19 2.99 (0.99%)
filter q17 453.66 444.82 8.84 (1.99%)
filter q18 1901.54 1946.70 -45.16 (-2.32%)
filter zonemap-node 94.62 86.57 8.05 (9.30%)
filter zonemap-node-lhs-cast 94.83 86.53 8.30 (9.60%)
filter zonemap-rel 5514.33 5743.59 -229.26 (-3.99%)
fixed_size_expr_evaluator q07 554.63 544.45 10.18 (1.87%)
fixed_size_expr_evaluator q08 769.32 778.80 -9.48 (-1.22%)
fixed_size_expr_evaluator q09 769.01 774.16 -5.15 (-0.67%)
fixed_size_expr_evaluator q10 246.38 238.02 8.36 (3.51%)
fixed_size_expr_evaluator q11 241.31 237.01 4.29 (1.81%)
fixed_size_expr_evaluator q12 239.48 236.82 2.66 (1.12%)
fixed_size_expr_evaluator q13 1480.16 1503.38 -23.22 (-1.54%)
fixed_size_seq_scan q23 120.82 119.91 0.90 (0.75%)
join q29 593.76 662.44 -68.69 (-10.37%)
join q30 1389.88 1289.55 100.33 (7.78%)
join q31 5.48 7.01 -1.53 (-21.84%)
ldbc_snb_ic q35 415.55 406.73 8.82 (2.17%)
ldbc_snb_ic q36 142.31 126.12 16.20 (12.84%)
ldbc_snb_is q32 5.65 5.16 0.49 (9.41%)
ldbc_snb_is q33 12.82 15.12 -2.29 (-15.16%)
ldbc_snb_is q34 1.61 1.61 -0.00 (-0.19%)
multi-rel multi-rel-large-scan 2041.69 1612.54 429.15 (26.61%)
multi-rel multi-rel-lookup 17.86 21.53 -3.66 (-17.02%)
multi-rel multi-rel-small-scan 101.91 99.58 2.33 (2.34%)
order_by q25 145.77 131.52 14.25 (10.83%)
order_by q26 456.41 460.60 -4.20 (-0.91%)
order_by q27 1415.52 1435.41 -19.89 (-1.39%)
scan_after_filter q01 175.60 174.04 1.55 (0.89%)
scan_after_filter q02 165.90 157.98 7.92 (5.01%)
shortest_path_ldbc100 q37 76.61 84.91 -8.31 (-9.78%)
shortest_path_ldbc100 q38 441.34 418.99 22.35 (5.34%)
shortest_path_ldbc100 q39 65.80 63.00 2.80 (4.44%)
shortest_path_ldbc100 q40 460.24 514.13 -53.89 (-10.48%)
var_size_expr_evaluator q03 2072.51 2102.75 -30.23 (-1.44%)
var_size_expr_evaluator q04 2212.90 2268.57 -55.68 (-2.45%)
var_size_expr_evaluator q05 2568.19 2699.13 -130.93 (-4.85%)
var_size_expr_evaluator q06 1347.63 1345.60 2.04 (0.15%)
var_size_seq_scan q19 1474.04 1479.78 -5.75 (-0.39%)
var_size_seq_scan q20 2633.17 2767.61 -134.44 (-4.86%)
var_size_seq_scan q21 2301.76 2317.91 -16.15 (-0.70%)
var_size_seq_scan q22 133.08 130.39 2.69 (2.07%)

Copy link

Benchmark Result

Master commit hash: 71f31389315b87c2c8cf2229727c59f20cac20b5
Branch commit hash: 592156bc3105e8a408f373d2df3f9bb0561e64aa

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 649.30 640.85 8.45 (1.32%)
aggregation q28 11381.91 11570.46 -188.55 (-1.63%)
filter q14 128.55 124.63 3.92 (3.15%)
filter q15 130.53 130.56 -0.02 (-0.02%)
filter q16 304.15 307.18 -3.03 (-0.99%)
filter q17 450.56 452.14 -1.58 (-0.35%)
filter q18 1908.03 1898.32 9.70 (0.51%)
filter zonemap-node 86.31 86.60 -0.29 (-0.34%)
filter zonemap-node-lhs-cast 87.20 88.91 -1.71 (-1.92%)
filter zonemap-rel 5617.49 5520.38 97.11 (1.76%)
fixed_size_expr_evaluator q07 545.36 541.80 3.56 (0.66%)
fixed_size_expr_evaluator q08 763.67 760.14 3.53 (0.46%)
fixed_size_expr_evaluator q09 758.17 760.29 -2.13 (-0.28%)
fixed_size_expr_evaluator q10 241.08 238.62 2.46 (1.03%)
fixed_size_expr_evaluator q11 236.34 232.45 3.89 (1.67%)
fixed_size_expr_evaluator q12 234.83 231.73 3.10 (1.34%)
fixed_size_expr_evaluator q13 1468.95 1471.10 -2.15 (-0.15%)
fixed_size_seq_scan q23 118.45 119.27 -0.82 (-0.69%)
join q29 638.83 627.74 11.08 (1.77%)
join q30 1317.21 1338.87 -21.66 (-1.62%)
join q31 5.84 5.81 0.02 (0.40%)
ldbc_snb_ic q35 415.95 446.22 -30.27 (-6.78%)
ldbc_snb_ic q36 136.34 118.61 17.72 (14.94%)
ldbc_snb_is q32 2.53 2.04 0.48 (23.65%)
ldbc_snb_is q33 12.19 13.41 -1.22 (-9.09%)
ldbc_snb_is q34 1.54 1.55 -0.01 (-0.58%)
multi-rel multi-rel-large-scan 1756.33 1824.78 -68.45 (-3.75%)
multi-rel multi-rel-lookup 32.41 31.54 0.87 (2.75%)
multi-rel multi-rel-small-scan 82.45 85.89 -3.44 (-4.01%)
order_by q25 130.67 127.15 3.52 (2.77%)
order_by q26 452.90 449.53 3.37 (0.75%)
order_by q27 1390.93 1394.39 -3.46 (-0.25%)
scan_after_filter q01 174.10 170.53 3.57 (2.10%)
scan_after_filter q02 154.65 160.25 -5.60 (-3.49%)
shortest_path_ldbc100 q37 82.64 91.20 -8.55 (-9.38%)
shortest_path_ldbc100 q38 459.15 491.79 -32.64 (-6.64%)
shortest_path_ldbc100 q39 60.33 59.86 0.47 (0.78%)
shortest_path_ldbc100 q40 532.82 546.92 -14.10 (-2.58%)
var_size_expr_evaluator q03 2041.80 2056.88 -15.08 (-0.73%)
var_size_expr_evaluator q04 2256.05 2217.98 38.07 (1.72%)
var_size_expr_evaluator q05 2667.92 2583.63 84.29 (3.26%)
var_size_expr_evaluator q06 1317.89 1334.42 -16.53 (-1.24%)
var_size_seq_scan q19 1453.77 1456.96 -3.19 (-0.22%)
var_size_seq_scan q20 2629.88 2566.20 63.68 (2.48%)
var_size_seq_scan q21 2296.98 2283.56 13.43 (0.59%)
var_size_seq_scan q22 127.27 127.12 0.15 (0.12%)

Copy link

Benchmark Result

Master commit hash: 71f31389315b87c2c8cf2229727c59f20cac20b5
Branch commit hash: 9a9841ae8f3f11a5013915736eac5c15d270d228

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 644.08 640.85 3.23 (0.50%)
aggregation q28 11360.68 11570.46 -209.78 (-1.81%)
filter q14 127.82 124.63 3.19 (2.56%)
filter q15 125.35 130.56 -5.21 (-3.99%)
filter q16 301.95 307.18 -5.23 (-1.70%)
filter q17 445.45 452.14 -6.69 (-1.48%)
filter q18 1926.56 1898.32 28.24 (1.49%)
filter zonemap-node 87.77 86.60 1.18 (1.36%)
filter zonemap-node-lhs-cast 86.67 88.91 -2.24 (-2.52%)
filter zonemap-rel 5523.39 5520.38 3.01 (0.05%)
fixed_size_expr_evaluator q07 547.46 541.80 5.67 (1.05%)
fixed_size_expr_evaluator q08 763.79 760.14 3.66 (0.48%)
fixed_size_expr_evaluator q09 765.48 760.29 5.19 (0.68%)
fixed_size_expr_evaluator q10 241.71 238.62 3.09 (1.30%)
fixed_size_expr_evaluator q11 236.41 232.45 3.96 (1.70%)
fixed_size_expr_evaluator q12 238.97 231.73 7.24 (3.12%)
fixed_size_expr_evaluator q13 1492.43 1471.10 21.33 (1.45%)
fixed_size_seq_scan q23 115.98 119.27 -3.29 (-2.76%)
join q29 637.04 627.74 9.29 (1.48%)
join q30 1423.23 1338.87 84.35 (6.30%)
join q31 6.72 5.81 0.90 (15.56%)
ldbc_snb_ic q35 423.65 446.22 -22.57 (-5.06%)
ldbc_snb_ic q36 151.09 118.61 32.47 (27.38%)
ldbc_snb_is q32 4.39 2.04 2.35 (114.72%)
ldbc_snb_is q33 14.14 13.41 0.73 (5.44%)
ldbc_snb_is q34 1.43 1.55 -0.12 (-7.64%)
multi-rel multi-rel-large-scan 1917.89 1824.78 93.11 (5.10%)
multi-rel multi-rel-lookup 8.05 31.54 -23.50 (-74.49%)
multi-rel multi-rel-small-scan 88.98 85.89 3.09 (3.60%)
order_by q25 129.99 127.15 2.84 (2.23%)
order_by q26 451.22 449.53 1.69 (0.38%)
order_by q27 1395.98 1394.39 1.59 (0.11%)
scan_after_filter q01 177.36 170.53 6.84 (4.01%)
scan_after_filter q02 155.77 160.25 -4.49 (-2.80%)
shortest_path_ldbc100 q37 86.67 91.20 -4.53 (-4.96%)
shortest_path_ldbc100 q38 488.96 491.79 -2.83 (-0.57%)
shortest_path_ldbc100 q39 61.80 59.86 1.94 (3.24%)
shortest_path_ldbc100 q40 488.93 546.92 -57.99 (-10.60%)
var_size_expr_evaluator q03 2048.27 2056.88 -8.61 (-0.42%)
var_size_expr_evaluator q04 2269.79 2217.98 51.81 (2.34%)
var_size_expr_evaluator q05 2658.12 2583.63 74.49 (2.88%)
var_size_expr_evaluator q06 1318.09 1334.42 -16.33 (-1.22%)
var_size_seq_scan q19 1454.17 1456.96 -2.79 (-0.19%)
var_size_seq_scan q20 2556.80 2566.20 -9.40 (-0.37%)
var_size_seq_scan q21 2276.88 2283.56 -6.68 (-0.29%)
var_size_seq_scan q22 127.39 127.12 0.27 (0.21%)

@SterlingT3485
Copy link
Collaborator Author

Let's also add some tests.

Tests are added via c api

src/main/database.cpp Show resolved Hide resolved
test/c_api/database_test.cpp Outdated Show resolved Hide resolved
src/common/file_system/local_file_system.cpp Outdated Show resolved Hide resolved

// Cleanup: Remove directories after the test
std::filesystem::remove_all("/tmp/test2");
std::filesystem::remove_all("/tmp/dbHome/test2");
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also add those test cases:

  1. Delete directories with edge cases.
  2. Test whether it works as expected on windows since windows paths can have a mixture of '\' and '/'.
    For example: homedir: "C:\Desktop\dir"
    Path to delete: "C:\Desktop/dir/test1"
  3. Test whether we error in wild card pattern paths. (e.g. path to delete: '/tmp/kuzu*.test')
  4. Test when we delete a path with homedirectory symbol (e.g. path to delete "~/test")

Copy link
Collaborator Author

@SterlingT3485 SterlingT3485 Nov 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added more tests, we do not support wild card patter for safety.

Copy link

Benchmark Result

Master commit hash: d12c21fa11921b4e2574589148024a87397d909f
Branch commit hash: 554dde6a1152fc71512adf543e187939ab886202

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 652.88 644.15 8.73 (1.36%)
aggregation q28 12081.14 11832.18 248.97 (2.10%)
filter q14 135.67 126.85 8.82 (6.95%)
filter q15 135.87 122.88 12.98 (10.57%)
filter q16 311.33 301.19 10.14 (3.37%)
filter q17 454.89 446.48 8.41 (1.88%)
filter q18 1928.54 1921.70 6.84 (0.36%)
filter zonemap-node 95.38 86.36 9.02 (10.44%)
filter zonemap-node-lhs-cast 95.41 87.13 8.28 (9.50%)
filter zonemap-rel 5490.00 5340.33 149.67 (2.80%)
fixed_size_expr_evaluator q07 583.96 574.34 9.62 (1.68%)
fixed_size_expr_evaluator q08 812.33 808.46 3.88 (0.48%)
fixed_size_expr_evaluator q09 813.38 803.73 9.65 (1.20%)
fixed_size_expr_evaluator q10 250.07 243.03 7.05 (2.90%)
fixed_size_expr_evaluator q11 242.24 233.89 8.36 (3.57%)
fixed_size_expr_evaluator q12 239.21 231.25 7.96 (3.44%)
fixed_size_expr_evaluator q13 1497.67 1484.48 13.19 (0.89%)
fixed_size_seq_scan q23 124.13 121.87 2.26 (1.86%)
join q29 643.01 608.43 34.58 (5.68%)
join q30 1433.79 1342.06 91.73 (6.83%)
join q31 7.22 3.84 3.38 (87.93%)
ldbc_snb_ic q35 403.32 420.02 -16.70 (-3.98%)
ldbc_snb_ic q36 113.62 132.33 -18.71 (-14.14%)
ldbc_snb_is q32 4.98 3.63 1.35 (37.16%)
ldbc_snb_is q33 13.09 11.80 1.29 (10.98%)
ldbc_snb_is q34 1.60 1.48 0.12 (7.75%)
multi-rel multi-rel-large-scan 1691.89 1648.60 43.29 (2.63%)
multi-rel multi-rel-lookup 6.24 5.78 0.46 (7.94%)
multi-rel multi-rel-small-scan 101.82 108.82 -7.00 (-6.43%)
order_by q25 140.63 132.70 7.94 (5.98%)
order_by q26 455.14 454.14 1.00 (0.22%)
order_by q27 1476.52 1457.04 19.48 (1.34%)
scan_after_filter q01 176.92 170.97 5.95 (3.48%)
scan_after_filter q02 164.84 156.19 8.65 (5.54%)
shortest_path_ldbc100 q37 83.09 90.62 -7.52 (-8.30%)
shortest_path_ldbc100 q38 483.25 468.18 15.07 (3.22%)
shortest_path_ldbc100 q39 60.45 60.57 -0.13 (-0.21%)
shortest_path_ldbc100 q40 562.23 481.30 80.93 (16.82%)
var_size_expr_evaluator q03 2057.94 2114.30 -56.36 (-2.67%)
var_size_expr_evaluator q04 2239.64 2281.78 -42.13 (-1.85%)
var_size_expr_evaluator q05 2603.80 2641.80 -38.01 (-1.44%)
var_size_expr_evaluator q06 1332.49 1324.95 7.54 (0.57%)
var_size_seq_scan q19 1460.16 1499.36 -39.20 (-2.61%)
var_size_seq_scan q20 2555.42 2444.44 110.98 (4.54%)
var_size_seq_scan q21 2282.69 2368.72 -86.03 (-3.63%)
var_size_seq_scan q22 127.02 127.77 -0.75 (-0.59%)

Copy link

Benchmark Result

Master commit hash: d12c21fa11921b4e2574589148024a87397d909f
Branch commit hash: 8638714ccbee370132fa9ba42848f8ccd8206986

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 644.09 644.15 -0.05 (-0.01%)
aggregation q28 11404.47 11832.18 -427.71 (-3.61%)
filter q14 125.13 126.85 -1.72 (-1.36%)
filter q15 125.65 122.88 2.76 (2.25%)
filter q16 303.47 301.19 2.28 (0.76%)
filter q17 446.94 446.48 0.46 (0.10%)
filter q18 1929.76 1921.70 8.06 (0.42%)
filter zonemap-node 86.65 86.36 0.29 (0.34%)
filter zonemap-node-lhs-cast 91.82 87.13 4.69 (5.39%)
filter zonemap-rel 5380.77 5340.33 40.45 (0.76%)
fixed_size_expr_evaluator q07 578.34 574.34 4.00 (0.70%)
fixed_size_expr_evaluator q08 808.96 808.46 0.50 (0.06%)
fixed_size_expr_evaluator q09 807.14 803.73 3.41 (0.42%)
fixed_size_expr_evaluator q10 241.13 243.03 -1.90 (-0.78%)
fixed_size_expr_evaluator q11 235.26 233.89 1.37 (0.59%)
fixed_size_expr_evaluator q12 231.05 231.25 -0.19 (-0.08%)
fixed_size_expr_evaluator q13 1486.44 1484.48 1.97 (0.13%)
fixed_size_seq_scan q23 116.54 121.87 -5.33 (-4.37%)
join q29 638.76 608.43 30.33 (4.99%)
join q30 1449.77 1342.06 107.70 (8.03%)
join q31 6.99 3.84 3.15 (82.05%)
ldbc_snb_ic q35 412.01 420.02 -8.01 (-1.91%)
ldbc_snb_ic q36 123.85 132.33 -8.47 (-6.40%)
ldbc_snb_is q32 4.72 3.63 1.08 (29.88%)
ldbc_snb_is q33 13.42 11.80 1.63 (13.79%)
ldbc_snb_is q34 1.70 1.48 0.22 (14.53%)
multi-rel multi-rel-large-scan 1754.89 1648.60 106.29 (6.45%)
multi-rel multi-rel-lookup 15.47 5.78 9.69 (167.52%)
multi-rel multi-rel-small-scan 82.70 108.82 -26.11 (-24.00%)
order_by q25 132.47 132.70 -0.22 (-0.17%)
order_by q26 446.94 454.14 -7.20 (-1.59%)
order_by q27 1447.37 1457.04 -9.67 (-0.66%)
scan_after_filter q01 171.02 170.97 0.05 (0.03%)
scan_after_filter q02 155.63 156.19 -0.56 (-0.36%)
shortest_path_ldbc100 q37 92.17 90.62 1.56 (1.72%)
shortest_path_ldbc100 q38 455.16 468.18 -13.02 (-2.78%)
shortest_path_ldbc100 q39 62.03 60.57 1.46 (2.40%)
shortest_path_ldbc100 q40 545.79 481.30 64.49 (13.40%)
var_size_expr_evaluator q03 2046.66 2114.30 -67.64 (-3.20%)
var_size_expr_evaluator q04 2226.55 2281.78 -55.22 (-2.42%)
var_size_expr_evaluator q05 2530.67 2641.80 -111.14 (-4.21%)
var_size_expr_evaluator q06 1344.43 1324.95 19.47 (1.47%)
var_size_seq_scan q19 1447.64 1499.36 -51.72 (-3.45%)
var_size_seq_scan q20 2428.23 2444.44 -16.21 (-0.66%)
var_size_seq_scan q21 2281.98 2368.72 -86.74 (-3.66%)
var_size_seq_scan q22 128.88 127.77 1.10 (0.86%)

Copy link

Benchmark Result

Master commit hash: d12c21fa11921b4e2574589148024a87397d909f
Branch commit hash: 1347906c271397f1e14c23c73ddb83e2a0f3a6ab

Query Group Query Name Mean Time - Commit (ms) Mean Time - Master (ms) Diff
aggregation q24 642.67 644.15 -1.47 (-0.23%)
aggregation q28 12001.65 11832.18 169.47 (1.43%)
filter q14 127.95 126.85 1.10 (0.87%)
filter q15 126.53 122.88 3.65 (2.97%)
filter q16 302.30 301.19 1.12 (0.37%)
filter q17 442.94 446.48 -3.55 (-0.79%)
filter q18 1934.03 1921.70 12.33 (0.64%)
filter zonemap-node 87.65 86.36 1.29 (1.49%)
filter zonemap-node-lhs-cast 86.99 87.13 -0.13 (-0.15%)
filter zonemap-rel 5471.98 5340.33 131.66 (2.47%)
fixed_size_expr_evaluator q07 577.11 574.34 2.77 (0.48%)
fixed_size_expr_evaluator q08 805.36 808.46 -3.10 (-0.38%)
fixed_size_expr_evaluator q09 805.69 803.73 1.96 (0.24%)
fixed_size_expr_evaluator q10 240.37 243.03 -2.66 (-1.09%)
fixed_size_expr_evaluator q11 232.82 233.89 -1.06 (-0.45%)
fixed_size_expr_evaluator q12 230.22 231.25 -1.03 (-0.45%)
fixed_size_expr_evaluator q13 1481.62 1484.48 -2.86 (-0.19%)
fixed_size_seq_scan q23 117.96 121.87 -3.91 (-3.21%)
join q29 614.89 608.43 6.46 (1.06%)
join q30 1429.51 1342.06 87.44 (6.52%)
join q31 8.18 3.84 4.34 (112.96%)
ldbc_snb_ic q35 406.52 420.02 -13.50 (-3.21%)
ldbc_snb_ic q36 119.48 132.33 -12.84 (-9.71%)
ldbc_snb_is q32 5.85 3.63 2.22 (61.14%)
ldbc_snb_is q33 14.30 11.80 2.50 (21.23%)
ldbc_snb_is q34 1.16 1.48 -0.33 (-22.06%)
multi-rel multi-rel-large-scan 1677.74 1648.60 29.14 (1.77%)
multi-rel multi-rel-lookup 5.74 5.78 -0.05 (-0.80%)
multi-rel multi-rel-small-scan 84.26 108.82 -24.56 (-22.57%)
order_by q25 132.34 132.70 -0.35 (-0.27%)
order_by q26 471.05 454.14 16.91 (3.72%)
order_by q27 1455.02 1457.04 -2.02 (-0.14%)
scan_after_filter q01 169.79 170.97 -1.18 (-0.69%)
scan_after_filter q02 155.37 156.19 -0.82 (-0.52%)
shortest_path_ldbc100 q37 86.00 90.62 -4.61 (-5.09%)
shortest_path_ldbc100 q38 467.04 468.18 -1.14 (-0.24%)
shortest_path_ldbc100 q39 59.08 60.57 -1.49 (-2.47%)
shortest_path_ldbc100 q40 519.14 481.30 37.84 (7.86%)
var_size_expr_evaluator q03 2062.56 2114.30 -51.74 (-2.45%)
var_size_expr_evaluator q04 2261.76 2281.78 -20.02 (-0.88%)
var_size_expr_evaluator q05 2598.52 2641.80 -43.29 (-1.64%)
var_size_expr_evaluator q06 1323.85 1324.95 -1.11 (-0.08%)
var_size_seq_scan q19 1471.40 1499.36 -27.96 (-1.86%)
var_size_seq_scan q20 2529.21 2444.44 84.77 (3.47%)
var_size_seq_scan q21 2306.60 2368.72 -62.12 (-2.62%)
var_size_seq_scan q22 127.47 127.77 -0.30 (-0.24%)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: Check whether the path to delete is a subdirectory of kuzu database path
2 participants