Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix #170: Re-create FTS Search table with newer tokenizer #357

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Fix #170: Re-create FTS Search table with newer tokenizer #357

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bc5ddeb
Upgrade to FTS4 with unicode61
TWiStErRob Oct 21, 2023
3587f17
Downgrade to buggy behavior to support API 21-29 too.
TWiStErRob Oct 21, 2023
575508b
Huh, comments are part of the schema
TWiStErRob Oct 24, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions android/database/src/main/assets/MagicHomeInventory.schema.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -673,8 +673,12 @@ CREATE VIEW Item_Path AS
join Category c ON i.category = c._id
;


CREATE VIRTUAL TABLE Search USING FTS3 (
-- FTS4 available since Android 3.0 (API 11) SQLite 3.7.4: https://www.sqlite.org/releaselog/3_7_4.html
-- Current minimum is Android 5.0 (API 21) SQLite 3.8.6: https://www.sqlite.org/releaselog/3_8_6.html,
-- which means "The unicode61 tokenizer is now included in FTS4 by default."
CREATE VIRTUAL TABLE Search USING FTS4 (
-- https://www.twisterrob.net/blog/2023/10/sqlite-unicode61-remove-diacritics-2.html
tokenize=unicode61 'remove_diacritics=1',
name,
location
);
Expand Down
16 changes: 16 additions & 0 deletions android/database/src/main/assets/MagicHomeInventory.upgrade.6.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
-- Context: https://github.com/TWiStErRob/net.twisterrob.inventory/issues/170
-- FTS3 supported only ASCII characters, FTS4 supports unicode when used with unicode61 tokenizer.

-- Recreate the Search table with new FTS4 format.
DROP TABLE IF EXISTS Search;
CREATE VIRTUAL TABLE Search USING FTS4 (
-- https://www.twisterrob.net/blog/2023/10/sqlite-unicode61-remove-diacritics-2.html
tokenize=unicode61 'remove_diacritics=1',
name,
location
Comment on lines +9 to +10
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include description in same upgrade?

);

-- Re-insert all data into the new table.
INSERT INTO Search_Refresher (_id) SELECT _id FROM Item;
-- Compact database indexes for faster search experience.
INSERT INTO Search(Search) VALUES('optimize');
Comment on lines +13 to +16
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Measure how much this costs to make sure users won't run out of disk space.

Copy link
Owner Author

@TWiStErRob TWiStErRob Oct 21, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

on 34

Upgraded (189 ms) database v5 to v6, step 5 to 6: v5(68)::3.39.2@/data/user/0/net.twisterrob.inventory.debug/databases/MagicHomeInventory

2 changes: 1 addition & 1 deletion android/database/src/main/java/net/twisterrob/inventory/android/content/Database.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ public Database(Context context) {
@VisibleForTesting
public Database(Context hostContext, Resources resources) {
super(resources);
m_helper = new DatabaseOpenHelper(hostContext, NAME, 5, BuildConfig.DEBUG) {
m_helper = new DatabaseOpenHelper(hostContext, NAME, 6, BuildConfig.DEBUG) {
@Override public void onConfigure(SQLiteDatabase db) {
super.onConfigure(db);
// CONSIDER enabling auto_vacuum=INCREMENTAL as it speeds up a large delete a lot
Expand Down
Loading