Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for entry of contracted braille via a braille keyboard #2439

Closed
nvaccessAuto opened this issue Jun 12, 2012 · 27 comments
Closed

Support for entry of contracted braille via a braille keyboard #2439

nvaccessAuto opened this issue Jun 12, 2012 · 27 comments
Assignees
Labels
component/braille enhancement GoogleWork p3 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority
Milestone

Comments

@nvaccessAuto
Copy link

Reported by jteh on 2012-06-12 06:11
(Spun off #808.)

Supporting input of contracted braille is much more complicated than computer braille, as we need to determine when to translate a chunk of input, the user experience before translation of a chunk occurs, etc.
Blocked by #808

@nvaccessAuto
Copy link
Author

Comment 2 by ragb on 2012-07-08 14:47
I believe contracted braille input is really realy hard to do, specially expecting a good user experience. I haven't seen any good support for portuguese contracted braille input anywhere. I don't know english braille in detail but I think we may have similar problems such as:

  • The same dot pattern (or multiple dot cells pattern) meaning completly different things. I.G. in portuguese, the pattern 126-3 might mean opening parentises or the characters "ê.". In portuguese the "o" character (135) is the same as the greater-then character.... With contractations the mess is even greater. So... a good experience, seems something very relative :).
  • Special signs (such as number-sign, caps sign, etc.) also mean stuff such as "#",, "_"...
  • Noone standardized and adapted braille to the needs and cenarious that came with tecnology and computers in particular. So many times there are no rules to follow, neither we don't know what to code neither users know what to expect many times. On braille input either then 8 dots, everone events their own rules (compare braille input in osx, in Jaws, or in brltty)....

Anyway, this is realy something that I'd like to have in NVDA (or anywhere else), but is something not trivial, in my opinion.

@nvaccessAuto
Copy link
Author

Comment 3 by Ahiiron on 2012-07-12 17:35
Just voicing my support for this. It would be a very painful/complex thing to do right, but man, it would rule. Then I can narrow down my display I want to purchase to ones equipped with a Braille keyboard.

@nvaccessAuto
Copy link
Author

Comment 4 by MHameed on 2012-07-13 13:23
I am slowly working on getting liblouis shaped up in terms of testability, correct tables etc.
backtranslation is working but has several problems, but I am hoping that issues will be fixed so that nvda and orca can use the backtranslation features to provide the requested support in the Medium term.

@nvaccessAuto
Copy link
Author

Comment 5 by nvdakor on 2012-07-13 16:36
Hi,
In that case, I think we should work on a new branch, as this is something that'll require some extensive testing by end users.

@nvaccessAuto
Copy link
Author

Comment 6 by ragb on 2012-12-04 00:55
Hi,

When implementing #808 I tried to use a literary braille table for back-translation. Of course the results were poor, since the goal for that ticket was computer braille input. In the meantime, I came with the following simple idea that might be a good starting point for this. (this requires familiarity with the work beeing developed on #808 and the !brailleInput branch).

Instead of translating a character as soon as a braille keyboard command is entered, we can back-translated the braille dots only when a full word is entered, that is when a space bar is pressed (this can be argued). Moreover, if no space bar is pressed after a while, we can assume that the input is done and the back-translation should be sent to the operating system anyway (this requires a timer). This way we can translate full words (or more characters at a time) instead of just one. The test being entered is not spoken or shown in braille until is sent, but when typing braille one can't read it at the same time anyway.

Steps for integrating this in #808 would be something like:

  1. In brailleInput.BrailleInputHandler} create a buffer and a timer. When some braille pattern either tan space is entered add it to the buffer and reset the timer.
  2. If space is entered, flush the buffer (back-translate and send to Windows, including the space character), and stop the timer (we don't need it while nothing is on the buffer).
  3. If no space is entered but the timer comes to zero, flush the buffer anyway.

I implemented a scatch prototype (I need to merge it with new !brailleInput changes) and it seems to work reasonably with english grade 1 braille. With portuguese is a nightmare :(. I'm using 1 second for the described timer, but I believe this value most be configurable.

I supose this is simliar to what VoiceOver on the Mac is doing, but I have no way to confirm it. However, I believe we can do much better and refine this simple algorithm, specially if liblouis evolves in back-translation.

@nvaccessAuto
Copy link
Author

Comment 7 by jteh on 2012-12-04 01:18
This is a good proof of concept. However, the timeout approach has been very harshly criticised by iOS users because a user doesn't think of the timeout and may take longer. A configurable timeout helps to mitigate this to some extent, but it's still not ideal. Also, not being able to read what has been entered could be bad for long words. These are the difficult questions i don't have answers for, but they need to be considered.

@nvaccessAuto
Copy link
Author

Comment 8 by jteh on 2012-12-04 01:20
This might be obvious, but please keep any work on this in a separate branch from brailleInput. Thanks!

@nvaccessAuto
Copy link
Author

Comment 9 by ragb (in reply to comment 7) on 2012-12-04 12:58
Replying to jteh:

This is a good proof of concept. However, the timeout approach has been very harshly criticised by iOS users because a user doesn't think of the timeout and may take longer. A configurable timeout helps to mitigate this to some extent, but it's still not ideal.

I think there should be a way to stop the timeout, that is, only consider the word entered when the user presses space or enter. We could have too different options: either provide a command to stop the timer (some chord, per haps), or make possible to configure the timout to infinity (no timeout :)).

However that rings another problem. If the user is writing a word and types something by mistake, he should be able to delete characters (using dot7, maybe), and type the right patterns to finish the word. This obviously makes the need for showing the user what's being entered...

Also, not being able to read what has been entered could be bad for long words.

Yes. I can't come with a good solution to this. Do you think having a custom braille region showing when the user is entering a word is possible? That region would contain the input being entered (in braille) and is to be updated whenever the user enters or deletes a braille character. I don't know if this brings a good user experience or not, but that is what I can think of. It's somehow resembles input composition.

These are the difficult questions i don't have answers for, but they need to be considered.

Me neither. However, I believe with some brainstorming we can come with better solutions from most of what is available (i.g. IOS).

@nvaccessAuto
Copy link
Author

Comment 10 by nvdakor on 2012-12-04 15:06
Hi,
I think we might want to also consider how notetakers handle braille input as well:

  • From what I've seen with notetakers, they do instant translation on the fly based on the current contracted braille code in use. For example, if I want to type the word "testing" on my Apex (at least using its wordprocessor), I would type, "t, e, st sign". Then KeySoft reads, "t, e, st sign". Then I use read current word command and KeySoft reads "test". Then I type "ing sign", and KeySoft reads "ing". Then if I invoke read current word, KeySoft reads "testing".
  • The downside of iOS is that it doesn't handle contractions which are conscenants - that is, a word that ends in a conscenant which is a word when contraction rule is applied. For example, if I type the word "book", I might type "boo". Then I would pause for about a second, then type "k", but VoiceOver would insert the word "knowledge".
  • A problem with time-out approach would be that there might be a delay when a key is pressed and when the char is inserted (especially for Bluetooth displays and if the user's display is far from the computer). Also, this poses a problem for a number of foreign languages (in case if we wish to extend this concept to foreign contracted braille codes, which is something that international users would want to see in a future release".
    For this reason, I think we need to work with John, Mesar and other LibLouis people on perhaps implementing "smart back translation" based on incoming dots and whether a space surrounds such a dot. In order for this to work, we may want to use a braille rule (at least for English contracted braille codes) where many contractions comes after certain dots such as 4-6, 5 and so forth.
    For example, the algorithm could go something like:
  • Find out which dot was pressed on the braille keyboard.
  • If the dot corresponds to any one of the special contraction prefix, then display the actual dot on the braille display while withhelding the dot from the screen e.g. store the dot on a variable.
  • If the user presses dot 7, delete the dot from the braille display and set the var's value to "none" (in C++, it'll be "").
  • After the dot has been pressed, if the user types a letter or a sign corresponding to the contraction, then ask LibLouis to backtranslate the prefix dot and the contraction letter and send it to the screen.
    In doing so, some rules may need to be specified:
  • Take care of punctuation: in this case, we can check for presence of special characters such as space, line feeds, etc.
  • If a number sign has been entered, initially treat it as the contraction corresponding to the number sign "ble in U.S. grade 2". To enter number, one would type the number sign again, or type dots 3-4-5-6 with dot 7.
    Potential flaws:
  • What if a punctuation is inserted in the middle of numbers or letters?
  • In case we will be assuming that the keyboard has 8 buttonns; what if there are keyboards with only six?
    I think, for short term, asking the user to type dot 7 with dot patterns for punctuation or providing one or two extra commands for treating the next dot press as special char would work. Then based on the characters which surrounds it, the entered punctuation or symbol can be back translated correctly (mostly for brackets and quotation marks). Another approach would be to utilize the dot decision algorithm such that certain dot combinations (mostly dots 5-6) can be used to decide whether the next entered character is a letter or something else (or using dot 4 would work, but then we may get issues with accents).
    Just my thoughts.

@nvaccessAuto
Copy link
Author

Comment 11 by jteh (in reply to comment 10) on 2012-12-07 17:04
Replying to nvdakor:

  • From what I've seen with notetakers, they do instant translation on the fly based on the current contracted braille code in use. For example, if I want to type the word "testing" on my Apex (at least using its wordprocessor), I would type, "t, e, st sign". Then KeySoft reads, "t, e, st sign".

It's significantly simpler for notetakers to do this because the entire platform is custom designed. For a screen reader, there are three problems:

  1. We can't send the characters to the application as they're being entered, as any new Braille cell could change the translation completely. For example, if the user is typing "tgr" for "together", we can't send "tg" because it needs to be changed once the "r" and space is received.
  2. If we don't send characters to the application, they won't get spoken as they're being typed. (Can anyone tell me how this works on iOS? I imagine it doesn't speak what's been typed until the timeout.)
  3. If we don't send characters to the application, they don't appear on the screen or, more importantly, the Braille display. NVDA is designed to display text from applications (which is normally exactly what the user wants). We could display the entered dots separately, but it would be extremely difficult to integrate the dots being entered with the rest of the actual text. As Rui points out, this is more like input composition, which is normally displayed separately from the actual text.
    This is why iOS and other screen readers implement a timeout. They want to be able to send the text to the application (and thus show the text to the user) without waiting for the end of each word.

One other problem for NVDA is that liblouis can't tell us what individual special Braille cells mean. For example, it can't tell us that dots 3-4 is the "st" sign unless it's part of a full translation.

  • The downside of iOS is that it doesn't handle contractions which are conscenants - that is, a word that ends in a conscenant which is a word when contraction rule is applied. For example, if I type the word "book", I might type "boo". Then I would pause for about a second, then type "k", but VoiceOver would insert the word "knowledge".

That's because of the timeout.

  • A problem with time-out approach would be that ... this poses a problem for a number of foreign languages

Out of interest, why is a timeout more of a problem for some languages than others?

Your ideas about special dot decisions in liblouis would require some pretty significant changes both to liblouis itself and all of its tables. I'm not necessarily saying it's a bad idea, but it'd probably be a very long way off.

@dkager
Copy link
Collaborator

dkager commented Nov 25, 2015

If liblouis is to be used to do the backtranslation it may be a good idea to investigate how well it performs these days. I know the Dutch (nl-NL and nl-BE) tables aren't written for backtranslation and probably fail for the parts that use multi-pass rules. (Side question: why does NVDA disable the multi-pass facilities when forward translating?)
For UEB there is a lot of great work done by the APH that is pending review and possible inclusion in a future version of liblouis. This code too appears to deal only with forward translation. I don't know how much of the backtranslation functionality is covered by unit tests but this might be a good place to start.

@jcsteh
Copy link
Contributor

jcsteh commented Nov 26, 2015

If liblouis is to be used to do the backtranslation it may be a good idea to investigate how well it performs these days.

liblouis is all we have. I take your point, though. I guess if it's really that bad, we'll have to push for improvements to the tables and/or liblouis before we can release this.

(Side question: why does NVDA disable the multi-pass facilities when forward translating?)

Because multi-pass breaks the input/output position mapping. I have had grand ideas of completely rewriting that position mapping code for a while, but I've never gotten to it. Arend Arends contributed a patch for this, but it hasn't been integrated yet and I believe there are some outstanding concerns.

@dkager
Copy link
Collaborator

dkager commented Nov 26, 2015

The next release of liblouis is planned for next Monday and should include the patches contributed by Arend, Bue, etc.

I’d be happy to look into backtranslation for the Dutch tables, though if it requires extensive code changes it’ll have to wait till the new year.

From: James Teh [mailto:notifications@github.com]
Sent: Thursday, November 26, 2015 02:16
To: nvaccess/nvda nvda@noreply.github.com
Cc: Davy Kager mail@davykager.nl
Subject: Re: [nvda] Support for entry of contracted braille via a braille keyboard (#2439)

If liblouis is to be used to do the backtranslation it may be a good idea to investigate how well it performs these days.

liblouis is all we have. I take your point, though. I guess if it's really that bad, we'll have to push for improvements to the tables and/or liblouis before we can release this.

(Side question: why does NVDA disable the multi-pass facilities when forward translating?)

Because multi-pass breaks the input/output position mapping. I have had grand ideas of completely rewriting that position mapping code for a while, but I've never gotten to it. Arend Arends contributed http://www.freelists.org/post/liblouis-liblouisxml/Update-output-positions-during-multi-pass-forward-translation-only a patch for this liblouis/liblouis#133 , but it hasn't been integrated yet and I believe there are some outstanding concerns.


Reply to this email directly or view it on GitHub #2439 (comment) . https://github.com/notifications/beacon/AC9y8Y5Tk75wiMamPMzj_9dVTbbCMaXXks5pJlTMgaJpZM4GpUFG.gif

@derekriemer
Copy link
Collaborator

  1. select UEB
  2. type dots a 3456-1
  3. type space.
    a is actually typed rather than 1

@jcsteh
Copy link
Contributor

jcsteh commented Oct 12, 2016

a is actually typed rather than 1

My sincere apologies. I can in fact reproduce this now that I'm running liblouis 3. Previously, I was running an outdated liblouis. I never considered the possibility that they might have regressed it in liblouis 3. :(

Both this and the "hello!" case are due to bugs in the liblouis 3 UEB tables. According to this mailing list post, UEB backtranslation issues are being addressed at present. That post also requests people to report any backtranslation issues they encounter.

Unfortunately, we can't fix this. Translation is based only on the word you're typing, not text at the cursor. Doing otherwise would require that the input and output tables always be the same, which is not always desirable.

Could it be possible to make NVDA work in this case only if input == output?

No. Later cells can still change the text of earlier cells. So, if the previous word was "tg" and you type "r", it has to try to erase the "g" and insert "ogether". Things get even more complicated if the user drops the cursor into the middle of a word. And those examples are without even trying to come up with obscure edge cases.

As a user, I'd find it useful to force NVDA to translate with out forcing an enter or space.

The benefit of this is when entering something like brailleit4u or something with a number in a word.

Well, you can do that without ending translation; UEB does support that (table bugs notwithstanding). Regardless, I've implemented this now.

@LeonarddeR
Copy link
Collaborator

@jcsteh: Reported this earlier on IRC, but for refference.
When you select UEB1 as input table and enter dot 6, dot6 will stick on the braille display until you do another braille input gesture. This is expected of course, however it leads to slightly unexpected behaviour when you press dot 6 and than change your mind and start typing on the pc keyboard. Dot 6 will stay until the braille buffer until you switch windows or resume braille input.

@jcsteh
Copy link
Contributor

jcsteh commented Nov 3, 2016

Deferring to 2017.1 due to outstanding issues. See #6449 (comment) for details.

@feerrenrut
Copy link
Contributor

Consider fixing "Make Unified English Braille the default Braille code" #6952 as part of this issue.

@josephsl
Copy link
Collaborator

josephsl commented Mar 10, 2017 via email

jcsteh added a commit that referenced this issue Jun 22, 2017
jcsteh added a commit that referenced this issue Jul 7, 2017
…ues #2439, #6054, #6113, #6935)

Beyond the main code in brailleInput to support uncontracted/contracted input, this includes the following changes:
* The list of braille tables has been moved out of the braille module into a separate brailleTables module. Braille tables are now added with a function rather than directly adding them to the data structure. Aside from being necessary in order to specify and check whether a table is contracted, this also makes the data about tables more extensible in future.
* As the data structure for braille tables has now changed and is no longer ordered, this was a good opportunity to sort the list of tables alphabetically when displaying them to the user.
* Updated to liblouis master shortly after release 3.2.0. The post 3.2.0 commits include some important back-translation fixes for UEB.
* brailleInput is now notified when reverting config or changing config profiles. This is necessary because brailleInput now maintains some state when the input table is changed.
* brailleInput is now initialised before braille at startup. This is because braille depends on brailleInput to get the currently untranslated input.
* Dot7 and dot8 are now universally bound to braille input specific scripts for erase and enter. Any braille display drivers that had bindings for backspace/enter for braille input have been adjusted accordingly.
* In the User Guide, a "Braille" section has been added above "Application Specific Features". The "Braille Control Types and States" section has been moved to a sub-section of this, and a sub-section on Braille Input has been added.
* Added a Unicode braille input table.
@josephsl
Copy link
Collaborator

josephsl commented Dec 6, 2018 via email

@josephsl
Copy link
Collaborator

josephsl commented Dec 6, 2018

Hi,

Never mind about my last comment. Perhaps Adriani didn't see the issue was actually closed.

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/braille enhancement GoogleWork p3 https://github.com/nvaccess/nvda/blob/master/projectDocs/issues/triage.md#priority
Projects
None yet
Development

No branches or pull requests

7 participants