Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Editing Korean letters with NVDA and braille display issue. #5640

Closed
khsbory opened this issue Dec 30, 2015 · 27 comments
Closed

Editing Korean letters with NVDA and braille display issue. #5640

khsbory opened this issue Dec 30, 2015 · 27 comments
Assignees
Milestone

Comments

@khsbory
Copy link

khsbory commented Dec 30, 2015

Hello, I’m Hyongsop Kim in Korea.
In Notepad or another word program, if I type just English letters, there is no problem.
But if I type Korean letters, my braille display cannot be scrolled to prior or next line, just panning within same line.
If I go to prior or next line, I have to press keyboard arrow keys, not braille display scroll key.
Also in word program, if I type some Korean letters and type English letters, the English letters are not appeared in my braille display.
So if I check, I have to move keyboard arrow keys.

Please check and fix this issue.

Thank you.

@josephsl
Copy link
Collaborator

josephsl commented Jan 1, 2016

Hi,
Confirmed.
Technical: This is because of the fact that input composition window gets focused, and panning the display shows this window. Panning up and down has no effect due to focus moving to this composition window.
@michaelDCurran and @nishimotz: Is there anything (like a hack) we can do for braille users using East Asian languages who may experience this issue? Thanks.

@jcsteh
Copy link
Contributor

jcsteh commented Jan 4, 2016 via email

@josephsl
Copy link
Collaborator

josephsl commented Jan 4, 2016

Hi, eventually found the correct one…

When I press arrow keys, cursor moves as expected, closing the input composition window in the process.

From: James Teh [mailto:notifications@github.com]
Sent: Sunday, January 3, 2016 4:38 PM
To: nvaccess/nvda
Cc: josephsl
Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

Hmm. So what do the arrow keys do? Do they exit composition?

Also, I don't understand the issue with mixed Korean/English. I would
have thought that typing English characters should exit composition.

Btw, I think you want @michaelDCurran. :)


Reply to this email directly or view it on GitHub #5640 (comment) .Image removed by sender.

@khsbory
Copy link
Author

khsbory commented Jan 4, 2016

The meaning of mixed Korean and English is that starting with some Korean letters and in the middle of Korean letters, typing some English.
And as Joseph says, in the middle of the typing Korean, moving computer keyboard arrow keys work correctly.
But if I press braille panning scroll keys, the cursor moves only that line, doesn't move to next line.

Thank you.

김 형 섭 / Hyongsop Kim
(주)엔비전스 / 대리

서울시 종로구 가회동 1-29
1-29 Gahoe-dong, Jongno-gu, Seoul, Korea
T. +82 70 4900 0895 F. +82 2 313 3645 M. +82 10 5359 5996

http://dialogueinthedark.co.kr
http://cafe.naver.com/dialogueinthedark

-----Original Message-----
From: "josephsl"<notifications@github.com>
To: "nvaccess/nvda"<nvda@noreply.github.com>;
Cc: "khsbory"<hyongsop.kim@nhn.com>;
Sent: 2016-01-04 (월) 09:41:11
Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

Hi, eventually found the correct one…

When I press arrow keys, cursor moves as expected, closing the input composition window in the process.

From: James Teh [mailto:notifications@github.com]

Sent: Sunday, January 3, 2016 4:38 PM

To: nvaccess/nvda

Cc: josephsl

Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

Hmm. So what do the arrow keys do? Do they exit composition?

Also, I don't understand the issue with mixed Korean/English. I would

have thought that typing English characters should exit composition.

Btw, I think you want @michaelDCurran. :)

Reply to this email directly or view it on GitHub <">#5640 (comment)> .Image removed by sender.


Reply to this email directly or view it on GitHub.

@jcsteh
Copy link
Contributor

jcsteh commented Jan 4, 2016 via email

@khsbory
Copy link
Author

khsbory commented Jan 5, 2016

Hello, in the middle of Korean typing, if I press key that change to English, composition window is finished.
If you need more information, please let me know.

Thank you.

김 형 섭 / Hyongsop Kim
(주)엔비전스 / 대리

서울시 종로구 가회동 1-29
1-29 Gahoe-dong, Jongno-gu, Seoul, Korea
T. +82 70 4900 0895 F. +82 2 313 3645 M. +82 10 5359 5996

http://dialogueinthedark.co.kr
http://cafe.naver.com/dialogueinthedark

-----Original Message-----
From: "James Teh"<notifications@github.com>
To: "nvaccess/nvda"<nvda@noreply.github.com>;
Cc: "khsbory"<hyongsop.kim@nhn.com>;
Sent: 2016-01-04 (월) 12:45:55
Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

I understood that, but what I'm querying is whether the composition

window disappears when English characters are typed. I imagined it

would, but it'd be good to have someone confirm this.


Reply to this email directly or view it on GitHub.

@jcsteh
Copy link
Contributor

jcsteh commented Jan 5, 2016 via email

@khsbory
Copy link
Author

khsbory commented Jan 6, 2016

Yes, if I open the new document and type just English, no problem.
But if I type Korean, next all English characters are not dsplayed in my braille.
In order to read, I have to press arrow key in my computer keyboard, not braille display.

Thank you.

김 형 섭 / Hyongsop Kim
(주)엔비전스 / 대리

서울시 종로구 가회동 1-29
1-29 Gahoe-dong, Jongno-gu, Seoul, Korea
T. +82 70 4900 0895 F. +82 2 313 3645 M. +82 10 5359 5996

http://dialogueinthedark.co.kr
http://cafe.naver.com/dialogueinthedark

-----Original Message-----
From: "James Teh"<notifications@github.com>
To: "nvaccess/nvda"<nvda@noreply.github.com>;
Cc: "khsbory"<hyongsop.kim@nhn.com>;
Sent: 2016-01-06 (수) 07:51:34
Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

So when you type English characters, the composition window disappears,

but NVDA doesn't recognise this and continues to display just the

composition in braille?


Reply to this email directly or view it on GitHub.

@jcsteh
Copy link
Contributor

jcsteh commented Jan 7, 2016

Also in word program, if I type some Korean letters and type English letters, the English letters are not appeared in my braille display.

Is this fixed in the latest snapshot for the "next" branch? I'm guessing this is #4145.

@khsbory
Copy link
Author

khsbory commented Jan 9, 2016

Sorry for the late reply.
As you said, I updated to Next version, 01/08/2016 released.
But unfortunately, the issue is not solved.

If you need more testing, please let me know.

김 형 섭 / Hyongsop Kim
(주)엔비전스 / 대리

서울시 종로구 가회동 1-29
1-29 Gahoe-dong, Jongno-gu, Seoul, Korea
T. +82 70 4900 0895 F. +82 2 313 3645 M. +82 10 5359 5996

http://dialogueinthedark.co.kr
http://cafe.naver.com/dialogueinthedark

-----Original Message-----
From: "James Teh"<notifications@github.com>
To: "nvaccess/nvda"<nvda@noreply.github.com>;
Cc: "khsbory"<hyongsop.kim@nhn.com>;
Sent: 2016-01-07 (목) 10:27:58
Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

Also in word program, if I type some Korean letters and type English letters, the English letters are not appeared in my braille display.

Is this fixed in the latest snapshot for the "next" branch? I'm guessing this is #4145.


Reply to this email directly or view it on GitHub.

@jcsteh
Copy link
Contributor

jcsteh commented Jan 11, 2016 via email

@josephsl
Copy link
Collaborator

Hi, Confirmed: when I type English, switch to Korean, type something and then switch back to English, I can’t see English characters on my BrailleNote (using it as a braille display). Thanks.

From: James Teh [mailto:notifications@github.com]
Sent: Sunday, January 10, 2016 4:02 PM
To: nvaccess/nvda nvda@noreply.github.com
Cc: josephsl joseph.lee22590@gmail.com
Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

@josephsl, can you reproduce the mixed Korean/Englihs issue in Word? I
don't understand why you can't see the English characters on the display
if the composition window disappears.


Reply to this email directly or view it on GitHub #5640 (comment) .

@jcsteh
Copy link
Contributor

jcsteh commented Jan 11, 2016 via email

@josephsl
Copy link
Collaborator

Here it is (may not be formatted correctly:

name: u'Microsoft Word Document'

role: ROLE_EDITABLETEXT

states: STATE_MULTILINE, STATE_FOCUSABLE, STATE_FOCUSED

isFocusable: True

hasFocus: True

Python object: <NVDAObjects.Dynamic_IAccessibleWordDocumentWindowNVDAObject object at 0x09E22570>

Python class mro: (<class 'NVDAObjects.Dynamic_IAccessibleWordDocumentWindowNVDAObject'>, <class 'NVDAObjects.IAccessible.IAccessible'>, <class 'NVDAObjects.window.winword.WordDocument'>, <class 'NVDAObjects.behaviors.EditableTextWithoutAutoSelectDetection'>, <class 'editableText.EditableTextWithoutAutoSelectDetection'>, <class 'NVDAObjects.behaviors.EditableText'>, <class 'editableText.EditableText'>, <class 'NVDAObjects.window.Window'>, <class 'NVDAObjects.NVDAObject'>, <class 'baseObject.ScriptableObject'>, <class 'baseObject.AutoPropertyObject'>, <type 'object'>)

description: None

location: (0, 172, 1349, 534)

value: None

appModule: <'appModuleHandler' (appName u'winword', process ID 5052) at address 9c520d0>

appModule.productName: u'Microsoft Office 2013'

appModule.productVersion: u'15.0.4779.1001'

TextInfo: <class 'NVDAObjects.window.winword.WordDocumentTextInfo'>

windowHandle: 461392

windowClassName: u'_WwG'

windowControlID: 0

windowStyle: 1442840576

windowThreadID: 1856

windowText: u'Microsoft Word Document'

displayText: u''

IAccessibleObject: <POINTER(IAccessible) ptr=0xb262f80 at 9e26620>

IAccessibleChildID: 0

IAccessible event parameters: windowHandle=461392, objectID=-4, childID=0

IAccessible accName: u'Microsoft Word Document'

IAccessible accRole: ROLE_SYSTEM_CLIENT

IAccessible accState: STATE_SYSTEM_FOCUSED, STATE_SYSTEM_FOCUSABLE, STATE_SYSTEM_VALID (1048580)

IAccessible accDescription: None

IAccessible accValue: None

From: James Teh [mailto:notifications@github.com]
Sent: Sunday, January 10, 2016 4:21 PM
To: nvaccess/nvda nvda@noreply.github.com
Cc: josephsl joseph.lee22590@gmail.com
Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

Can you provide the devInfo for the focus after typing English
characters please?


Reply to this email directly or view it on GitHub #5640 (comment) .

@jcsteh
Copy link
Contributor

jcsteh commented Jan 11, 2016 via email

@khsbory
Copy link
Author

khsbory commented Jan 11, 2016

Thank you Joseph, for helping me to explain detaily. If I type Korean, after all characters including English, numbers, other characters are not displayed on my braile display. So even just one word that Korean typed, all characters are not displayed. Thank you.

김 형 섭 / Hyongsop Kim
(주)엔비전스 / 대리

서울시 종로구 가회동 1-29
1-29 Gahoe-dong, Jongno-gu, Seoul, Korea
T. +82 70 4900 0895 F. +82 2 313 3645 M. +82 10 5359 5996

http://dialogueinthedark.co.kr
http://cafe.naver.com/dialogueinthedark

-----Original Message-----
From: "James Teh"<notifications@github.com>
To: "nvaccess/nvda"<nvda@noreply.github.com>;
Cc: "khsbory"<hyongsop.kim@nhn.com>;
Sent: 2016-01-11 (월) 10:41:42
Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

Okay. So the composition window is definitely disappearing. Do you see

other text on the line or only the Korean characters you just composed?

And you're saying only the English characters are missing? Are the

English characters always missing then?


Reply to this email directly or view it on GitHub.

@josephsl
Copy link
Collaborator

Hi,

English characters disappear when typing them after typing Korean.

STR:

  1. Type soething in English.
  2. Switch to Korean and type something in Korean.
  3. Switch back to English.
    Expected: English characters are displayed.
    Actual: English characters are missing.
  4. Type some more English letters.
  5. Switch to Korean and type something in Korean.
    Result: The missing English characters are then shown.
    Thanks.

@jcsteh
Copy link
Contributor

jcsteh commented Jan 11, 2016

When the English characters are missing, if you alt+tab out and back in again, do they appear? Can you give me a quick key sequence to get some Korean characters for testing?

@josephsl
Copy link
Collaborator

Hi,
Yes, English characters appear when I press Alt+TAB to switch from and to Word.

Some keystrokes I used:

  1. After installing Korean input method, press Windows+SPACE to switch to Korean (Windows 8 and later; I'm using Windows 10).
  2. Press right Alt to switch between Korean (native) and English (alphanumeric) inputs. Then type:

English: Hello (then no space)
Korean: xptmxld (testing)
English (no space after Korean): Hi

Note that braille output table should be Korean also.
Thanks.

@michaelDCurran
Copy link
Member

I have located the underlying issue:
in NVDAHelper.handleCompositionEnd: focus is moved back to the parent of the composition (the document) with an executeEvent. However, IAccessibleHandler, when routing caret events to the focus, only checks lastQueuedFocusObject. LastQueuedFocusObject is most likely still the composition object.

The easiest way to fix this is to use queueEvent instead of executeEvent. However, there may be side-effects to do with timing with input composition handling. We will have to just see.

A safer way might be to have executeEvent take an optional keyword argument, say "fromQueue" which would only be set to true when coming from the queue (i.e. not called directly by outside code). If fromQueue is False, executeEvent itself should update lastQueuedFocusObject as from an external point of view, the call is really the equivalent to queueEvent except that it happens straight away.

Any thoughts @jcsteh?

@michaelDCurran
Copy link
Member

A simple way to describe this issue, if not noted already, is that once composition ends, although the braille display again shows the document content, it no longer updates due to typed characters. Selection detection is also broken.

@jcsteh
Copy link
Contributor

jcsteh commented Jan 18, 2016

in NVDAHelper.handleCompositionEnd: focus is moved back to the parent of the composition (the document) with an executeEvent. However, IAccessibleHandler, when routing caret events to the focus, only checks lastQueuedFocusObject. LastQueuedFocusObject is most likely still the composition object.

But handleInputCompositionStart also uses executeEvent, so why would lastQueuedFocusObject be the composition object? And if it isn't, what is it then?

The easiest way to fix this is to use queueEvent instead of executeEvent. However, there may be side-effects to do with timing with input composition handling. We will have to just see.

I'm actually a bit worried about the fact that we use executeEvent at all. executeEvent isn't supposed to be called from a background thread. That means speech and braille could be happening in the background thread and we only guarantee that those need to work in the main thread. Also, there could be race conditions with focus events between the two threads.

A safer way might be to have executeEvent take an optional keyword argument, say "fromQueue" which would only be set to true when coming from the queue (i.e. not called directly by outside code).

If there's some special reason we need to use executeEvent for input composition, we could also just set lastQueuedFocusObject directly before calling it. Obviously, we shouldn't be tweaking eventHandler internals like that, but it's probably safer to deal with a special case internally rather than exposing something publicly which might be frequently misunderstood. Yes, anyone could tweak lastQueuedFocusObject, but if they do, we can say "you tweaked stuff you shouldn't have; you're on your own".

@michaelDCurran
Copy link
Member

Please test this try build which hopefully fixes the issue:
http://www.nvaccess.org/files/nvda/try/i5640/nvda_snapshot_try-i5640-11798,c38150f.exe

@khsbory
Copy link
Author

khsbory commented Jan 20, 2016

Hello, thank you very much for your fixing this issue.
All issues that braille scroll problem and no braille appear problem when I type Korean and English mixed are resolved.
Thank you again.

김 형 섭 / Hyongsop Kim
(주)엔비전스 / 대리

서울시 종로구 가회동 1-29
1-29 Gahoe-dong, Jongno-gu, Seoul, Korea
T. +82 70 4900 0895 F. +82 2 313 3645 M. +82 10 5359 5996

http://dialogueinthedark.co.kr
http://cafe.naver.com/dialogueinthedark

-----Original Message-----
From: "Michael Curran"<notifications@github.com>
To: "nvaccess/nvda"<nvda@noreply.github.com>;
Cc: "khsbory"<hyongsop.kim@nhn.com>;
Sent: 2016-01-20 (수) 15:14:03
Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

Please test this try build which hopefully fixes the issue:

http://www.nvaccess.org/files/nvda/try/i5640/nvda_snapshot_try-i5640-11798,c38150f.exe


Reply to this email directly or view it on GitHub.

@josephsl
Copy link
Collaborator

Hi Mick, I can also confirm the issue is resolved. Thanks. Now the question is if this fix broke anything major…

From: khsbory [mailto:notifications@github.com]
Sent: Wednesday, January 20, 2016 1:27 AM
To: nvaccess/nvda nvda@noreply.github.com
Cc: josephsl joseph.lee22590@gmail.com
Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

Hello, thank you very much for your fixing this issue.
All issues that braille scroll problem and no braille appear problem when I type Korean and English mixed are resolved.
Thank you again.

김 형 섭 / Hyongsop Kim
(주)엔비전스 / 대리

서울시 종로구 가회동 1-29
1-29 Gahoe-dong, Jongno-gu, Seoul, Korea
T. +82 70 4900 0895 F. +82 2 313 3645 M. +82 10 5359 5996

http://dialogueinthedark.co.kr
http://cafe.naver.com/dialogueinthedark

-----Original Message-----
From: "Michael Curran"<notifications@github.com>
To: "nvaccess/nvda"<nvda@noreply.github.com>;
Cc: "khsbory"<hyongsop.kim@nhn.com>;
Sent: 2016-01-20 (수) 15:14:03
Subject: Re: [nvda] Editing Korean letters with NVDA and braille display issue. (#5640)

Please test this try build which hopefully fixes the issue:

http://www.nvaccess.org/files/nvda/try/i5640/nvda_snapshot_try-i5640-11798,c38150f.exe


Reply to this email directly or view it on GitHub.


Reply to this email directly or view it on GitHub #5640 (comment) .

@nvaccessAuto
Copy link

Incubated in cd543f6.

@jcsteh jcsteh added this to the 2016.1 milestone Feb 1, 2016
@nishimotz
Copy link
Contributor

Sorry for slow response.
The use of Microsoft Japanese IME (or Office IME) and braille display causes similar issue, which still occurs the try build nvda_snapshot_try-i5640-11798,c38150f.exe.

Recently Japanese Team have confirmed that this issue happens with 2013.3jp and later versions.
(not yet tested with prior versions)

It should be filed as new issue, because Japanese IME may be something different from Korean IME.
Note that Japanese braille translation is only supported by NVDA JP and not yet supported by NVDA itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants