Speakup in userspace

Speakup in userspace

[Tyler Spivey]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I have some oppinions on this, and I'd like some input on how possible
this is. I'm not a very good C coder, but that's less important. I'm
hoping, if a user space screen reader is implemented, that it can be
extended; e.g. perl, python, etc. The problems with this seems to be:
1. How can a userspace program get extended key sequences such as caps
lock+i? Under Linux, I don't think that an API exists for this.

2. What about console output? There is /dev/vcs[a]x, but polling that
a few times a second would be inefficient if the system was under load.
The rest is relatively easy - Linux already has serial port devices if
needed, and also speech dispatcher communications. I see no reason why a
smaller subset of speakup (just an API to get keyboard/console
input/output) couldn't be included into the kernel - it probably
wouldn't change much, compared to the user-space adapter that would be
using that API. This is similar to what the Jupiter project is doing;
although that's all in-kernel.
any thoughts?
- - Tyler

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (FreeBSD)

iD8DBQFGeOZxTsjaYASMWKQRAt/WAKCsG0jwPECPii4+yQX6i4gO02UdXQCgmmuY
e5jsV1bQGCZd+8LLWjwTHTs=
=xOzu
-----END PGP SIGNATURE-----

Re: Speakup in userspace

[Luke Yelavich]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, Jun 20, 2007 at 06:33:54PM EST, Tyler Spivey wrote:
> 1. How can a userspace program get extended key sequences such as caps
> lock+i? Under Linux, I don't think that an API exists for this.

THis is possible, using the uinput module from the kernel.

> 2. What about console output? There is /dev/vcs[a]x, but polling that
> a few times a second would be inefficient if the system was under load.

BrlTTY uses this, and does so quite efficiently. As it is, if a user 
wants to run speakup and BrlTTY, there could be a problem with both 
programs wanting to share the same devices.

THere has in fact been discussion on the BrlTTY mailing list about 
extending BrlTTY to have speech/screen reader via keyboard 
functionality. Patches would be accepted for BrlTTY to do this.
- -- 
Luke Yelavich
GPG key: 0xD06320CE 
	 (http://www.themuso.com/themuso-gpg-key.txt)
Email & MSN: themuso@themuso.com
Jabber: themuso@jabber.org.au
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFGePnGjVefwtBjIM4RAnMNAJ43ASbKihiYyojkrDJLvfC4WdMCWgCgjCXf
slLR/lzF7/uHbYGy9Cpgl5E=
=xftW
-----END PGP SIGNATURE-----

Re: Speakup in userspace

[Samuel Thibault]

Luke Yelavich, le Wed 20 Jun 2007 19:56:22 +1000, a écrit :
> -----BEGIN PGP SIGNED MESSAGE-----
> On Wed, Jun 20, 2007 at 06:33:54PM EST, Tyler Spivey wrote:
> > 1. How can a userspace program get extended key sequences such as caps
> > lock+i? Under Linux, I don't think that an API exists for this.
> 
> THis is possible, using the uinput module from the kernel.

And it is tedious: you have to grab the keyboard, and re-simulate
keypresses you don't interpret.

> > 2. What about console output? There is /dev/vcs[a]x, but polling that
> > a few times a second would be inefficient if the system was under load.
> 
> BrlTTY uses this,

For simulating keypresses, yes, but not for stealing key combinations.

Samuel

Re: Speakup in userspace

[Luke Yelavich]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, Jun 20, 2007 at 07:59:52PM EST, Samuel Thibault wrote:
> Luke Yelavich, le Wed 20 Jun 2007 19:56:22 +1000, a écrit :
> > -----BEGIN PGP SIGNED MESSAGE-----
> > On Wed, Jun 20, 2007 at 06:33:54PM EST, Tyler Spivey wrote:
> > > 1. How can a userspace program get extended key sequences such as caps
> > > lock+i? Under Linux, I don't think that an API exists for this.
> > 
> > THis is possible, using the uinput module from the kernel.
> 
> And it is tedious: you have to grab the keyboard, and re-simulate
> keypresses you don't interpret.

Ok. Was unaware of this.

> > > 2. What about console output? There is /dev/vcs[a]x, but polling that
> > > a few times a second would be inefficient if the system was under load.
> > 
> > BrlTTY uses this,
> 
> For simulating keypresses, yes, but not for stealing key combinations.

I was referring to reading the console, which is what I think Tyler was 
referring to.
- -- 
Luke Yelavich
GPG key: 0xD06320CE 
	 (http://www.themuso.com/themuso-gpg-key.txt)
Email & MSN: themuso@themuso.com
Jabber: themuso@jabber.org.au
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFGeP1CjVefwtBjIM4RAhUsAKDxhDCdDMqUMkB/ExeZ4yJqLUtwbACggx2l
UDG2iVT0iEMWnYE21e8q+aQ=
=Sgb8
-----END PGP SIGNATURE-----

Re: Speakup in userspace

[Samuel Thibault]

Luke Yelavich, le Wed 20 Jun 2007 20:11:14 +1000, a écrit :
> > > > 2. What about console output? There is /dev/vcs[a]x, but polling that
> > > > a few times a second would be inefficient if the system was under load.
> > > 
> > > BrlTTY uses this,
> > 
> > For simulating keypresses, yes, but not for stealing key combinations.
> 
> I was referring to reading the console, which is what I think Tyler was 
> referring to.

Ooops, sorry, yes. brltty polls every 40ms, and it seems to be fine.
That said, I have in my todo-list a request for i-notification that
would make this fare more efficient.

Samuel

Re: Speakup in userspace

[Doug Sutherland]

A few useful references

http://linuxconsole.sourceforge.net/input/input.html

Using the Input Subsystem - No matter how many buttons an input device
has or how many kinds of events it can generate, you can now work with
it from user space.

http://www.linuxjournal.com/article/6429

Here is part one of that article series
http://www.linuxjournal.com/article/6396

Here is a useful way to browse around in the input driver source
http://www.gelato.unsw.edu.au/lxr/source/drivers/input/?a=i386

This POC /dev/input/event/* keylogger is interesting
http://archives.neohapsis.com/archives/sf/linux/2005-q3/0065.html

This is an interesting simple example driver code for input driver
http://kernel.org/pub/linux/kernel/people/landley/kdocs/Documentation/input/input-programming.txt

This is old but useful due to its description of the bigger picture
of keyboard scancodes and related stuff
http://www.linuxjournal.com/article/1080

As for console output, just as with the Input Driver redesign, it is
changing quicky and they are trying to clean up the architecture.
I suppose the first thing would be to see how the console is
changing and where it's heading architecture wise. I have a lot
of interest in this for other projects and will be doing some
investigation.

One thing I have in mind is implementing what will basically be the
equivalent of a VT100 terminal using a small microcontroller, where
input is an attached keyboard and output is speech. Since there
already is serial console, a means to implement a terminal with
keyboard input and speech output would be quite useful. Rather
than connecting a machine to serial port for console, you'd plug
in a dongle which would have both a keyboard port and an
output port for speech, or possibly even speech hardware in the
actual dongle. I'll share some more info on this idea a bit later.

  -- Doug

Re: Speakup in userspace

[Samuel Thibault]

Hi,

Doug Sutherland, le Wed 20 Jun 2007 06:20:51 -0500, a écrit :
> A few useful references
> 
> http://linuxconsole.sourceforge.net/input/input.html

Yes and no.  What we want is stealing some keypresses, but not all of
them.  The way to achieve this with input is currently to grab the
physical device, and re-emulate the keypresses we don't want in a uinput
device, which is quite tedious.

Samuel

Re: Speakup in userspace

[Doug Sutherland]

Samuel said:
What we want is stealing some keypresses, but not all of
them.  The way to achieve this with input is currently to grab the
physical device, and re-emulate the keypresses we don't want in a uinput
device, which is quite tedious.

Fair enough. But it would seem that the components that make up a 
specialized system like this need to be decoupled rather than monolithic,
for many reasons, one being allowance for alternative modalities. What 
would be very useful I think, would be if those "stolen" keypresses could
land in a  /dev/input event queue. This decouples the input device from 
what is done what those stolen keypresses. It would also allow for some
other modality, not PC keyboard, to generate those same events by 
some other means. Consider someone who cannot type for whatever
reason, who needs an alternative interface other than the keyboard.

The Input Driver may be tedious but it's uniform and modular. If we are 
talking about potential user space versus kernel space, then it would make 
some sense to dissect the overall functionality into layers and separate 
them, then figure out what needs to remain in the kernel, ideally as 
somewhat "atomic" functionality, probably as several drivers instead of 
one, then think about how in user space an application might use those. 
The approach of putting it all in the kernel has quite a few detriments, 
probably the largest being the need for constant maintenance as the kernel 
changes. One small example, although it may be tedious to steal key
presses and then re-implement them, once that is done as a separate 
driver on its own, then very little should change from kernel to kernel 
version. One the other end of things, console output, there is even 
greater need for layering and atomicity of function, since what used 
to be a serial port is functionally the same now but the protocols and 
hardware to implement it (usb serial) is completely different. 

It might be a good idea to think in terms of what I would call 
vertical protoype if we want to explore user space versus kernel.
What I mean by that is rather than thinking of how the entire new 
design would work, take a small cross-section of the whole and 
try re-designing a simple example that traverses from input to output. 
In commercial, in-house, or contract software this approach is 
essential (vertical prototype) to ensure that the architecture is 
sound before attempting to re-design the whole mess. 

The need to steal keypresses is not new or unique. It makes a lot
of sense to pay close attention to the re-architecture of linux input
method and console projects. Even if it's tedious or we don't like 
how some things are done, throwing it out and going direct to 
hardware digs a hole that makes it hardware specific to x86 and
most likely a never ending cycle of maintenance so it doesn't 
break as the kernel evolves. 

Thinking about the whole process in abstraction somewhat like
the model-view-controller design pattern would make sense.
Stealing the keypresses should be independent and atomically
abstracted from what those keypresses do.

  -- Doug

Re: Speakup in userspace

[Samuel Thibault]

Doug Sutherland, le Wed 20 Jun 2007 07:40:32 -0500, a écrit :
> The need to steal keypresses is not new or unique. It makes a lot
> of sense to pay close attention to the re-architecture of linux input
> method and console projects. Even if it's tedious or we don't like 
> how some things are done, throwing it out and going direct to 
> hardware digs a hole that makes it hardware specific to x86 and
> most likely a never ending cycle of maintenance so it doesn't 
> break as the kernel evolves. 

I didn't mean we should go hardware :) That's why I said "Yes and
No": Yes input is the way to go, but No it's not ready yet, we still
need a fine way to steal keypresses.

Samuel

Re: Speakup in userspace

[Christopher Moore]

This has been an interesting thread.  Regarding input devices, could a 
user space speakup be controlled from a external numeric keypad plugged 
into a usb port?  This would depend on whether we could identify the key 
presses as coming from the external keypad.  brltty has a stable 
approach for gethering console output.  So one of the missing pieces is 
a way to control speech output.  If key presses on an external keypad 
could be routed to speakup and not available to other applications, we 
might have a relatively simple solution.

Chris
Chris

Re: Speakup in userspace

[Samuel Thibault]

Christopher Moore, le Wed 20 Jun 2007 11:40:08 -0400, a écrit :
> This has been an interesting thread.  Regarding input devices, could a 
> user space speakup be controlled from a external numeric keypad plugged 
> into a usb port?  This would depend on whether we could identify the key 
> presses as coming from the external keypad.

As it would appear as another /dev/input/eventX device, yes.

Samuel

Re: Speakup in userspace

[Albert E. Sten-Clanton]

Can anybody tell me what "user space" means and where to read about it?  I don't recall finding the term in anything I've read so far.  Thanks!

Al

Re: Speakup in userspace

[Doug Sutherland]

Consider the linux that most of use to be a "protected mode" 
operating system as opposed to "real mode". Protected mode
allows access to things like virtual memory, multi-threading, 
and priviledge levels not available in real mode. Protected 
mode has been the standard on x86 PCs since the 80286.

A protected mode system segregates virtual memory into 
kernel space and user space. Kernel space is strictly reserved
for running the kernel, device drivers, and kernel extensions.
It is usually the case that kernel space memory is not swapped
to disk since that is much slower, which user space memory 
can be swapped to disk.

User space or "userland" processes cannot access the memory
of other processes, the basis of memory protection which 
makes linux very stable. Prior to win2k, the windows os was 
not a protected memory system, hence the freezing up or 
crashing of whole system from one bug in one driver or app.
A user space process, although restricted in memory access,
can request the kernel to map part of its memory onto its own
space, and can also access shared memory. 

The kernel space is the direct hardware access space along
with the management software that controls virtual memory,
DMA, threads, processes, etc. You have kernel processes 
and user processes. The kernel processes are supposed to 
be basic things like the direct interface to hardware. User
space is where applications run. So there is kernel space 
memory, threads, and processes, and user space memory, 
threads, and processes. 

Consider ALSA sound as an example. It's in the kernel but
it's also not in the kernel. There are kernel drivers and there
are user space libraries. The alsa-lib delegates sound control
to user space. This allows application developers to do all 
kinds of things without touching kernel code. The alsa-lib 
provides various functionality like software mixing, support
for the older OSS API, and user specific configuration, and
it is multi-thread safe, essential for complex audio programs.

Alsa may not be the best example, but the idea is separating
the core functionality from the application layer. Let's say I
create an API for writing text to a speech synth. The code 
that actually talks to the synth would ideally be abstracted 
from the API such that the identical programming interface
works for any synth using any protocol like serial or usb.
Some hardware may not implement all parts of the API but
where there are same functions the API should look the 
same. An example of a very well abstracted API is the 
Java API. It had to be done that way in order to make the
programs portable on different systems. I may be biased 
because I used to work there, but if you look at how much
work was done on abstraction it's the most impressively 
abstracted API around. I'm not talking about javascript, 
that's like a virus hehe. Unfortunately Sun wanted Java to 
be the answer to everything everywhere which it is not and
will never be, and Java, like many good ideas, has become
overly bloated and complex, although at least the various 
parts of it are separate APIs, and the compact versions 
like J2ME are still very efficient. They run on almost all 
phones now. There is a good reason for this. I wrote some
apps on blackberry and it was a breeze to do so. Compared
with doing it in C or ASM it's an entirely different world.

  -- Doug

Re: Speakup in userspace

[W. Nick Dotson]

Wow!  If I had you here as my mentor, I could learn that Linux stuff.  That just had to be the neatest bit of technical explanation I read in aeons...  I saved it 
in my Linux docs folder on my Windows machine...  (grin)

Nick

On Wed, 20 Jun 2007 13:53:35 -0500, Doug Sutherland wrote:

 Consider the linux that most of use to be a "protected mode" 
 operating system as opposed to "real mode". Protected mode
 allows access to things like virtual memory, multi-threading, 
 and priviledge levels not available in real mode. Protected 
 mode has been the standard on x86 PCs since the 80286.

 A protected mode system segregates virtual memory into 
 kernel space and user space. Kernel space is strictly reserved
 for running the kernel, device drivers, and kernel extensions.
 It is usually the case that kernel space memory is not swapped
 to disk since that is much slower, which user space memory 
 can be swapped to disk.

 User space or "userland" processes cannot access the memory
 of other processes, the basis of memory protection which 
 makes linux very stable. Prior to win2k, the windows os was 
 not a protected memory system, hence the freezing up or 
 crashing of whole system from one bug in one driver or app.
 A user space process, although restricted in memory access,
 can request the kernel to map part of its memory onto its own
 space, and can also access shared memory. 

 The kernel space is the direct hardware access space along
 with the management software that controls virtual memory,
 DMA, threads, processes, etc. You have kernel processes 
 and user processes. The kernel processes are supposed to 
 be basic things like the direct interface to hardware. User
 space is where applications run. So there is kernel space 
 memory, threads, and processes, and user space memory, 
 threads, and processes. 

 Consider ALSA sound as an example. It's in the kernel but
 it's also not in the kernel. There are kernel drivers and there
 are user space libraries. The alsa-lib delegates sound control
 to user space. This allows application developers to do all 
 kinds of things without touching kernel code. The alsa-lib 
 provides various functionality like software mixing, support
 for the older OSS API, and user specific configuration, and
 it is multi-thread safe, essential for complex audio programs.

 Alsa may not be the best example, but the idea is separating
 the core functionality from the application layer. Let's say I
 create an API for writing text to a speech synth. The code 
 that actually talks to the synth would ideally be abstracted 
 from the API such that the identical programming interface
 works for any synth using any protocol like serial or usb.
 Some hardware may not implement all parts of the API but
 where there are same functions the API should look the 
 same. An example of a very well abstracted API is the 
 Java API. It had to be done that way in order to make the
 programs portable on different systems. I may be biased 
 because I used to work there, but if you look at how much
 work was done on abstraction it's the most impressively 
 abstracted API around. I'm not talking about javascript, 
 that's like a virus hehe. Unfortunately Sun wanted Java to 
 be the answer to everything everywhere which it is not and
 will never be, and Java, like many good ideas, has become
 overly bloated and complex, although at least the various 
 parts of it are separate APIs, and the compact versions 
 like J2ME are still very efficient. They run on almost all 
 phones now. There is a good reason for this. I wrote some
 apps on blackberry and it was a breeze to do so. Compared
 with doing it in C or ASM it's an entirely different world.

   -- Doug

 _______________________________________________
 Speakup mailing list
 Speakup@braille.uwo.ca
 http://speech.braille.uwo.ca/mailman/listinfo/speakup

Re: Speakup in userspace

[Albert E. Sten-Clanton]

Thanks very much for the explanation below.

Al
----- Original Message ----- 
From: "Doug Sutherland" <doug@proficio.ca>
To: "Speakup is a screen review system for Linux." <speakup@braille.uwo.ca>
Sent: Wednesday, June 20, 2007 2:53 PM
Subject: Re: Speakup in userspace


> Consider the linux that most of use to be a "protected mode" 
> operating system as opposed to "real mode". Protected mode
> allows access to things like virtual memory, multi-threading, 
> and priviledge levels not available in real mode. Protected 
> mode has been the standard on x86 PCs since the 80286.
> 
> A protected mode system segregates virtual memory into 
> kernel space and user space. Kernel space is strictly reserved
> for running the kernel, device drivers, and kernel extensions.
> It is usually the case that kernel space memory is not swapped
> to disk since that is much slower, which user space memory 
> can be swapped to disk.
> 
> User space or "userland" processes cannot access the memory
> of other processes, the basis of memory protection which 
> makes linux very stable. Prior to win2k, the windows os was 
> not a protected memory system, hence the freezing up or 
> crashing of whole system from one bug in one driver or app.
> A user space process, although restricted in memory access,
> can request the kernel to map part of its memory onto its own
> space, and can also access shared memory. 
> 
> The kernel space is the direct hardware access space along
> with the management software that controls virtual memory,
> DMA, threads, processes, etc. You have kernel processes 
> and user processes. The kernel processes are supposed to 
> be basic things like the direct interface to hardware. User
> space is where applications run. So there is kernel space 
> memory, threads, and processes, and user space memory, 
> threads, and processes. 
> 
> Consider ALSA sound as an example. It's in the kernel but
> it's also not in the kernel. There are kernel drivers and there
> are user space libraries. The alsa-lib delegates sound control
> to user space. This allows application developers to do all 
> kinds of things without touching kernel code. The alsa-lib 
> provides various functionality like software mixing, support
> for the older OSS API, and user specific configuration, and
> it is multi-thread safe, essential for complex audio programs.
> 
> Alsa may not be the best example, but the idea is separating
> the core functionality from the application layer. Let's say I
> create an API for writing text to a speech synth. The code 
> that actually talks to the synth would ideally be abstracted 
> from the API such that the identical programming interface
> works for any synth using any protocol like serial or usb.
> Some hardware may not implement all parts of the API but
> where there are same functions the API should look the 
> same. An example of a very well abstracted API is the 
> Java API. It had to be done that way in order to make the
> programs portable on different systems. I may be biased 
> because I used to work there, but if you look at how much
> work was done on abstraction it's the most impressively 
> abstracted API around. I'm not talking about javascript, 
> that's like a virus hehe. Unfortunately Sun wanted Java to 
> be the answer to everything everywhere which it is not and
> will never be, and Java, like many good ideas, has become
> overly bloated and complex, although at least the various 
> parts of it are separate APIs, and the compact versions 
> like J2ME are still very efficient. They run on almost all 
> phones now. There is a good reason for this. I wrote some
> apps on blackberry and it was a breeze to do so. Compared
> with doing it in C or ASM it's an entirely different world.
> 
>   -- Doug
> 
> _______________________________________________
> Speakup mailing list
> Speakup@braille.uwo.ca
> http://speech.braille.uwo.ca/mailman/listinfo/speakup
> 
> 
> -- 
> No virus found in this incoming message.
> Checked by AVG Free Edition. 
> Version: 7.5.472 / Virus Database: 269.9.1/854 - Release Date: 6/19/2007 1:12 PM
> 
>

RE: Speakup in userspace

[Deborah Norling]

Maybe this is stupid, but if you need user-space screen access, couldn't you
just use Yasr? Can't its keymap be modified enough to mimic speakup?

I guess yasr is really pretty primitive compared to speakup; it can't track
highlighting for example. 

But maybe it would be easier, programatically to just improve yasr rather
than rewrite speakup.
--Debee