v2.4.4

Bug Fixes

• Identified root cause for #1662 to be recycling of TermFieldReaders that was causing illegal/incorrect access of several in-memory structures in certain scenarios. We've gone ahead and disabled this feature with #2117 + #2121 . Will work towards re-enabling in the near future once we've ironed out the several associated wrinkles.
• Introduced a guard rail with blevesearch/zapx#282 while performing vector search queries with pre-filtering to avoid hitting a panic when qualified docs do not hold valid vector fields.
• Fixed an issue while applying ivf_max_codes_pct in vector search requests involving pre-filtering which can cause reduction in recall - blevesearch/go-faiss#40

Vector search continues to require same version of faiss dynamic library (as with v2.4.3) to be built from blevesearch/faiss@b747c55a which is a modified version of v1.8.0

Milestone

• v2.4.4

v2.4.3

Bug Fixes

• Address a corner case with ivf_nprobe_pct query parameter (blevesearch/go-faiss#34)
• Several guard rails put in place to avoid array-out-of-bounds-access and divide-by-zero errors: (blevesearch/zapx#263, blevesearch/zapx#270, blevesearch/zapx#271, blevesearch/zapx#273) to overcome #1662
• Handling early exits/optimization for boolean queries (#2065)

Improvements

• Vector search requires faiss dynamic library to be built from blevesearch/faiss@b747c55a which is a modified version of v1.8.0
• Support for cosine similarity distance metric to normalize vectors before indexing/querying for nearest neighbor search (#2051)
• Support for selectivity filters as a pre-cursor to vector search (#2063)

Milestone

• v2.4.3

v2.4.2

Bug Fixes

• Set MaxSegmentFileSize for force merge when SingleSegmentMergePlanOptions goes into use (#2050)
• Protect code against any bolt failures (#2043)
• Fix size estimation for various field types (#2052)
• Address an out-of-bounds panic that could occur with zapx/v16 in the event of a single non-vector field (#2058)
  • blevesearch/zapx#252

Improvements

• Support new search params for vector search queries - [ivf_probe_pct, ivf_max_codes_pct] (#2049)

Milestone

• v2.4.2

v2.4.1

Bug Fixes

• Addressed a bug where ClientContextID was missing from SearchRequest when library used with vectors (#2014)
• Fix to a memory leak in vector query path (#2023)
• Fix to issue #2027, error in parsing a certain datetime syntax (#2030)
• Fix that mitigates a race between persister's stale segment removal and index copy operations (#2032)
• Nested field mapping determination was broken (#2031)

Improvements

• Vector search requires the faiss dynamic library to be built from blevesearch/faiss@d9db66a
• Support for new data type - vector_base64 which allows for interpreting vector float32s encoded as base64 following littleEndian byte ordering (#2012)
• Vector dimensionality support up to 4096 from 2048 (#2015)
• New vector index optimization - memory_efficient alongside pre-existing recall and latency (#2040)
• Better resource usage and efficiency in vector indexing (blevesearch/zapx#241, blevesearch/zapx#234)
• Delayed releasing of deserialized vector indexes to support better query throughput (#2016)

Milestone

• v2.4.1

v2.4.0

Bug Fixes

• Fixes to spanish accents normalization (#1957)
• Critical fix around facets merging which could've caused inconsistent results (#1946)

Improvements

• Support for vectors indexing and search
  • Please read: vectors.md for more details
  • With this we will now default to a new zap format: v16, which co-locates text and vector indexes within scorch segments. v16 will be able to interpret v15 zap segments.
  • Requires the faiss dynamic library to be built from blevesearch/faiss@7b119f4
• An empty query attribute within a SearchRequest will now be interpreted as a match_none (#2000)

Milestone

• v2.4.0

v2.3.10

Bug Fixes

• Fix format of date times displayed in search result, to not drop precision (#1868)

Improvements

• Upgraded to golang.org/x/[email protected] to overcome a vulnerability (#1845)
• Support for polish language analyzer - pl (#1825)
• Additional attribute in search response to determine partial/complete hits for match queries (#1848)
• Performance optimization to speed up documents processing per mapping definition (#1865)
• Support fuzziness factor for matchphrase, multiphrase & phrase queries (#1847)
• Geo shape polygon decode optimization w/ buffer re-use (#1864)
• Override-able Marshal/Unmarshal methods (#1880)
• Enhanced data time format support:
  • sanitizedgo that validates layout against supported golang constants (#1877)
  • Ability to apply custom datetime parsers to query criteria and facets (#1860, #1878)
  • Ability to parse date times in these formats - unix_micro, unix_sec, unix_milli, unix_nano (#1870)
  • Additional layouts support with percentstyle and isostyle (#1866)

Milestone

• v2.3.10

v2.3.9

Bug Fixes

• Fixes along recording IO bytes read for different query types (#1829)
  • Also changes the attribute name in the SearchResult construct from bytesRead to cost
• Fix incorrect facet counts when facetted queries are coupled with sorting over the same field (#1836)

Improvements

• Upgrade to RoaringBitmap/roaring@v1.2.3 (#1822)
• Upgrade to etcd-io/bblot@v1.3.7 (#1824)
• Consolidated some redundant work in a hot index path with #1832 which showed index time speed up by about 30% in certain situations especially while working with large documents.
• Upgrade to vellum@v1.0.10 which contains a minor optimization (#1838)

Milestone

• v2.3.9

v2.3.8

Bug Fixes

• Addresses panics that could occur within zap (with [email protected])
  • blevesearch/zapx#105
  • blevesearch/zapx#150
  • blevesearch/zapx#151
• Fixes analyzer inheritance for default mapping with #1807
• Addresses #1816 by fixing CLI's completion generation with #1817

Enhancements

• Introduces a new not-so-aggressive plural stemmer (stemmer_en_plural) for english with #1808

Milestone

• v2.3.8

v2.3.7

Bug Fixes

• Replaced deprecated io/ioutil functions with #1781
• Fixed situation where bytesRead will not appear in search response if query gets everything it needs from mmaped content with #1788
• Fixed a panic due to misalignment within chunkedContentCoder on 32-bit platforms with blevesearch/zapx#148 and #1791 (only applicable to zap@v15)
• A bot upgraded the golang.org/x/text version we use from 0.3.7 to 0.3.8 with #1792 🥇
• Include bytesWritten into bbolt (metadata for scorch) with #1774

Enhancements

• Support for improved rollback behavior by allowing sampling of snapshots based on a time interval with #1776 . Default behavior remains the same
• Changed the signature of scorch's AsyncErrorCallback with #1800 to provide more context. This is a breaking change.

Milestone

• v2.3.7

v2.3.6

Bug Fixes

• More efficient alignment for chunkedContentCoder (blevesearch/zapx#147)
• Addressed issue in indexing unigrams before punctuations in Japanese (cjk) (#1724)
• Addressed facets string output limitation for date ranges and numeric ranges reported by #1763 with #1764
• Fixed inaccurate IO stats accounting while loading segment metadata (#1769)

Enhancements

• New bytesRead attribute in search response (#1752, blevesearch/zapx#142)
  • Applicable to zapx v15 only
• New tests showcasing performance benchmarks for popular queries (#1755)

Milestone

• v2.3.6