enhancement(LorieView.java): improve input handling

[twaik]

Making LorieView handle korean/chinese/turkish/other languages requiring composition and composed symbols.
@knyipab can you please test this? It replaces #620 solution.

@813ethan @ilhan-athn7 @robertkirkman @hansm629 can you please test this too?

[hansm629]

@twaik
Here is the test result video.

There is still a delay when typing in Korean,
and there is also a symptom where the text is entered as if it is automatically completed.

SHANA.korean.input.mp4

[twaik]

The solution is created for software keyboard, I did not try to fix hardware keyboard.
Also in the case if you use "prefer scancodes when possible" input will be handled by fcitx or similar mechanism in linux, not by my code.

[hansm629]

@twaik
Does that mean testing should be conducted using the Android software keyboard instead of a USB or Bluetooth hardware keyboard?

[robertkirkman]

In Android Studio if I click the Debug button using this PR, then reproduce the crash, the debugger does not seem to print any backtrace, but this appears in adb logcat, when I type any character in Chromium inside a Termux:X11 app built from this PR using Gboard, and I can confirm that with this PR I am able to type "Ç" but Backspace makes the app close, but without this PR I cannot type "Ç" but backspace does not make the app close.

logcat snippet

2024-12-11 08:24:05.312 17579-17579 DEVICES                 com.termux.x11                       D  found device "hall" sources 0x80000000
2024-12-11 08:24:05.312 17579-17579 DEVICES                 com.termux.x11                       D  requesting stylus false
2024-12-11 08:24:05.312 17579-17579 DEVICES                 com.termux.x11                       D  external keyboard connected false
2024-12-11 08:24:05.345 17579-17579 ViewRootIm...nActivity] com.termux.x11                       I  Relayout returned: old=(0,0,1440,2960) new=(0,0,1440,2960) req=(1440,2960)0 dur=16 res=0x1 s={true 538304110592} ch=false
2024-12-11 08:24:10.778 17579-17579 LorieNative             com.termux.x11                       D  Parsing text: æ
2024-12-11 08:24:10.778 17579-17579 LorieNative             com.termux.x11                       D  Sending unicode event: æ (U+E6)
2024-12-11 08:24:12.181 17579-17579 AndroidRuntime          com.termux.x11                       D  Shutting down VM
2024-12-11 08:24:12.203 17579-17579 AndroidRuntime          com.termux.x11                       E  FATAL EXCEPTION: main
                                                                                                    Process: com.termux.x11, PID: 17579
                                                                                                    java.lang.IndexOutOfBoundsException: replace (0 ... -1) has end before start
                                                                                                    	at android.text.SpannableStringBuilder.checkRange(SpannableStringBuilder.java:1328)
                                                                                                    	at android.text.SpannableStringBuilder.replace(SpannableStringBuilder.java:523)
                                                                                                    	at android.text.SpannableStringBuilder.delete(SpannableStringBuilder.java:231)
                                                                                                    	at android.text.SpannableStringBuilder.delete(SpannableStringBuilder.java:38)
                                                                                                    	at com.termux.x11.LorieView$1.delete(LorieView.java:82)
                                                                                                    	at android.view.inputmethod.BaseInputConnection.deleteSurroundingText(BaseInputConnection.java:252)
                                                                                                    	at com.android.internal.view.IInputConnectionWrapper.executeMessage(IInputConnectionWrapper.java:450)
                                                                                                    	at com.android.internal.view.IInputConnectionWrapper$MyHandler.handleMessage(IInputConnectionWrapper.java:89)
                                                                                                    	at android.os.Handler.dispatchMessage(Handler.java:107)
                                                                                                    	at android.os.Looper.loop(Looper.java:237)
                                                                                                    	at android.app.ActivityThread.main(ActivityThread.java:8167)
                                                                                                    	at java.lang.reflect.Method.invoke(Native Method)
                                                                                                    	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:496)
                                                                                                    	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1100)
2024-12-11 08:24:12.302 17579-17579 Process                 com.termux.x11                       I  Sending signal. PID: 17579 SIG: 9
2024-12-11 08:24:12.403  1322-1841  InputDispatcher         pid-1322                             E  channel '8bb50eb com.termux.x11/com.termux.x11.MainActivity (server)' ~ Channel is unrecoverably broken and will be disposed!
2024-12-11 08:24:12.465  1322-3564  WindowManager           pid-1322                             E  win=Window{8bb50eb u0 com.termux.x11/com.termux.x11.MainActivity EXITING} destroySurfaces: appStopped=false win.mWindowRemovalAllowed=true win.mRemoveOnExit=true win.mViewVisibility=0 caller=com.android.server.wm.AppWindowToken.destroySurfaces:1249 com.android.server.wm.AppWindowToken.destroySurfaces:1230 com.android.server.wm.WindowState.onExitAnimationDone:5189 com.android.server.wm.-$$Lambda$01bPtngJg5AqEoOWfW3rWfV7MH4.accept:2 java.util.ArrayList.forEach:1262 com.android.server.wm.AppWindowToken.onAnimationFinished:3941 com.android.server.wm.AppWindowToken.commitVisibility:914 
2024-12-11 08:24:12.917   904-999   BufferQueueProducer     pid-904                              E  [SurfaceView - com.termux.x11/com.termux.x11.MainActivity@23442d7@9#0] dequeueBuffer: BufferQueue has been abandoned
�

termux-info of the testing device

Termux Variables:
TERMUX_APP_PACKAGE_MANAGER=apt
TERMUX_APP__APK_FILE=/data/app/com.termux-afP421Bfk35XjVh1_X0Sdw==/base.apk
TERMUX_APP__APK_RELEASE=F_DROID
TERMUX_APP__APP_VERSION_CODE=1020
TERMUX_APP__APP_VERSION_NAME=0.119.0-beta.1
TERMUX_APP__DATA_DIR=/data/user/0/com.termux
TERMUX_APP__IS_DEBUGGABLE_BUILD=false
TERMUX_APP__IS_INSTALLED_ON_EXTERNAL_STORAGE=false
TERMUX_APP__PACKAGE_NAME=com.termux
TERMUX_APP__PID=21866
TERMUX_APP__TARGET_SDK=28
TERMUX_VERSION=0.119.0-beta.1
TERMUX__SE_FILE_CONTEXT=u:object_r:app_data_file:s0:c33,c257,c512,c768
TERMUX__SE_INFO=untrusted:targetSdkVersion=28:complete
TERMUX__SE_PROCESS_CONTEXT=u:r:untrusted_app_27:s0:c33,c257,c512,c768
TERMUX__UID=10289
TERMUX__USER_ID=0
Packages CPU architecture:
aarch64
Subscribed repositories:
# sources.list
deb https://packages.termux.org/apt/termux-main stable main
# x11-repo (sources.list.d/x11.list)
deb https://packages.termux.org/apt/termux-x11 x11 main
# tur-repo (sources.list.d/tur.list)
deb https://tur.kcubeterm.com tur-packages tur tur-on-device tur-continuous
Updatable packages:
appstream/stable 1.0.4 aarch64 [upgradable from: 1.0.3]
clang/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
cmake/stable 3.31.2 aarch64 [upgradable from: 3.31.1]
code-server/tur-packages 4.95.3 aarch64 [upgradable from: 4.89.1]
command-not-found/stable 2.4.0-54 aarch64 [upgradable from: 2.4.0-52]
firefox/x11 132.0.2-1 aarch64 [upgradable from: 132.0.2]
gst-plugins-bad/stable 1.24.10 aarch64 [upgradable from: 1.24.9-1]
gst-plugins-base/stable 1.24.10 aarch64 [upgradable from: 1.24.9]
gst-plugins-good/stable 1.24.10 aarch64 [upgradable from: 1.24.9-1]
gstreamer/stable 1.24.10 aarch64 [upgradable from: 1.24.9]
json-glib/stable 1.10.6 aarch64 [upgradable from: 1.10.0]
libcompiler-rt/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
libdrm/stable 2.4.124 aarch64 [upgradable from: 2.4.123]
libllvm-static/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
libllvm/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
libopenmpt/stable 0.7.12 aarch64 [upgradable from: 0.7.11]
libpolly/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
librav1e/stable 0.7.1-2 aarch64 [upgradable from: 0.7.1-1]
libsoup3/stable 3.6.1 aarch64 [upgradable from: 3.4.4]
lld/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
lldb/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
llvm-tools/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
llvm/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
llvmgold/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
luanti/x11 1:5.10.0-4 aarch64 [upgradable from: 1:5.10.0-3]
mediainfo/stable 24.12 aarch64 [upgradable from: 24.11]
mesa-dev/stable 24.2.8 all [upgradable from: 24.2.7]
mesa/stable 24.2.8 aarch64 [upgradable from: 24.2.7]
mlir/stable 19.1.5 aarch64 [upgradable from: 19.1.4]
openal-soft/stable 1.24.1 aarch64 [upgradable from: 1.24.0]
openjpeg/stable 2.5.3 aarch64 [upgradable from: 2.5.2]
qt5-qtbase/x11 5.15.16 aarch64 [upgradable from: 5.15.14-2]
qt5-qtsvg/x11 5.15.16 aarch64 [upgradable from: 5.15.14-1]
qt5-qtx11extras/x11 5.15.16 aarch64 [upgradable from: 5.15.14-1]
ruby/stable 3.3.6 aarch64 [upgradable from: 3.3.5]
rust-std-aarch64-linux-android/stable 1.83.0-2 all [upgradable from: 1.82.0]
rust/stable 1.83.0-2 aarch64 [upgradable from: 1.82.0]
toxic/stable 0.16.0 aarch64 [upgradable from: 0.15.1]
tree-sitter/stable 0.24.5 aarch64 [upgradable from: 0.24.4]
unrar/stable 7.1.2 aarch64 [upgradable from: 7.1.1]
vim/stable 9.1.0900-1 aarch64 [upgradable from: 9.1.0850]
vulkan-headers/stable 1.4.303 all [upgradable from: 1.3.302]
vulkan-loader-generic/stable 1.4.303 aarch64 [upgradable from: 1.3.302]
vulkan-tools/stable 1.4.303 aarch64 [upgradable from: 1.3.302]
termux-tools version:
1.44.5
Android version:
10
Kernel build information:
Linux localhost 4.9.186-22990479 #1 SMP PREEMPT Thu Feb 24 18:21:21 KST 2022 aarch64 Android
Device manufacturer:
samsung
Device model:
SM-G960U
LD Variables:
LD_LIBRARY_PATH=
LD_PRELOAD=/data/data/com.termux/files/usr/lib/libtermux-exec.so
Installed termux plugins:
com.termux.api versionCode:51
com.termux.x11 versionCode:15

[twaik]

Does that mean testing should be conducted using the Android software keyboard instead of a USB or Bluetooth hardware keyboard?

Software keyboard only.

I can confirm that with this PR I am able to type "Ç" but Backspace makes the app close

Can you please try again?

[twaik]

@robertkirkman And what software keyboard layout did you use?

[twaik]

I mean I used gboard and it was fine on my device.

[twaik]

It lets us use cursor control, keyboard suggestions (with word replacings!) and some other stuff.

Screen_Recording_20241211_172139_TermuxX11.mp4

[twaik]

Korean input seems to be a bit more weird.

Screen_Recording_20241211_172818_TermuxX11.mp4

[hansm629]

@twaik
Should I also test this test with the option off?

Hardware keyboard scancodes workaround

Screen_Recording_20241212_005155_TermuxX11.mp4

[knyipab]

@twaik
Have you ever try to use backspace in Gboard English and other non-CJK langauges (with autocomplete and word suggestion)? The IME incorrectly think that it is working on some composed text that is yet to be commited, making backspace not functioning well.

This is the main reason why #620 needs to test the language used for Gboard and decide the input type dynamically. It would great if such behaviour can be suppressed.

[ilhan-athn7]

@813ethan @ilhan-athn7 @robertkirkman @hansm629 can you please test this too?

The cursor movement isn't usable on LatinIME
But this one works flawlessly on my end.

[twaik]

The cursor movement isn't usable on LatinIME

I did not find cursor control feature there. Probably it is not available there.

[robertkirkman]

@robertkirkman And what software keyboard layout did you use?

I used GBoard in Turkish mode, and now I have tested the current version, and the crash is not happening anymore in the same test, Backspace is now working. I believe the update you created fixed the problem.

Now I will describe a second problem that I noticed, which I believe I noticed happening at least once in both the original and the revised versions of this PR, and I am not sure how important this is because it seems like it only happens in Chromium and when I try to reproduce it in Firefox it never occurs, or at least I have not noticed it happening for approximately 20 minutes of testing firefox. Since Chromium is in tur-repo but Firefox is in x11-repo, I am not completely sure how high-priority this would be since it could be specifically just an issue with Chromium. What happens is, very occasionally, a symbol will appear that is different from the symbol that I touched, but only for one key stroke, and then the next touch of the same button produces the correct symbol.

Here is a video of that happening, at the timestamp in the video where it happened, 2:25, where for one key stroke, "é" became "í", and then "é" appeared on the next touch of the same button. https://youtu.be/jAVW-7wSjug?si=Fo8O9Hv0lQyf9yua&t=145

[ilhan-athn7]

The cursor movement isn't usable on LatinIME

I did not find cursor control feature there. Probably it is not available there.

screen-20241212-165829.mp4

[twaik]

It is not the same IME I have.

[813ethan]

seems fine with gboard chinese handwriting and cantonese input for now

logs like this appeared while typing tho, there seems to be one keysym log every character typed

(II) Added unknown keysym 0x10089ba to keycode 202
(II) Added unknown keysym 0x1005f97 to keycode 184
(II) Added unknown keysym 0x1005f88 to keycode 183

[813ethan]

the Deactivate special keys on additional key bar after each key press option is also not working on this pr