Re: [apache-plusplus] Process model ideas for C++ Apache.

[Michael Anderson <mka@redes.int.com.mx>]

Somehow we wandered off the topic of C++ Apache process model
to threading eccentricities ...

Dean Gaudet wrote:
> >This is where it's important to know what threading model
> >is in use... if these are all kernel-level threads, then this
> >definately hurts really hard. But if they're user-level
> >threads then context-switches are far less expensive.  The
> >memory burn is there either way (I'm worried about this in
> >threaded apache -- we have far far far too many 8k buffers
> >on the stack).

Ben Laurie wrote:
> OTOH, it may be worth the context switches to be able to limit
> the total number of threads. Suddenly this all makes sense to me.

My understanding of threads is that there are three basic
variations available without hacking your kernel (please correct
me if I'm wrong now - I haven't looked at this in a while):

1. kernel-level - heavy weight kernel scheduling. Examples are
sproc() (SGI) and clone() (LinuxThreads) which are really Unix
processes with shared data. The kernel can schedule these
across multiple CPUs.

2. user-level (fibers) - light wight application scheduling
within a single process. Examples are Provenzano's and Mueller's
pthreads packages. My understanding is that these cannot be
scheduled across multiple CPUs, since the kernel doesn't
understand the user-level implementation.

3. hybrid kernel/user-level - mid-weight kernel scheduling
within a single process. ?Example is Solaris threads (I'm not
sure of this?). These can be scheduled across multiple CPUs.

If fibers can not be spread across multiple CPUs, isn't there
a good possibility that a process with many fibers (say 300)
will be starving many of its fibers since the process is
competing for CPU (one CPU at a time) with all the other
processes in the system?
If this is the case, drastically reducing the number of
fibers per process may make sense and the division of
program functionality to reduce fiber count may look a lot
different even than what I proposed.

Dean Gaudet wrote: 
> > The apache API is actually already positioned to allow
> >explicit switching at various sequence points. In particular,
> >everything hangs off the request_rec structure. The
> >process_request_internal() function which goes through the
> >various stages of request processing is essentially just a
> >long list of sequence points. It would be possible to break
> >things up in the manner you're proposing right inside this
> >function. Something we should keep in mind while designing
> a new API...

Ben Laurie wrote:
> So the _really_ interesting question is, can we abstract it
> in such a way that we can decide whether to do start-to-end
> processing in a single thread, or whether to use pools of
> threads synchronized at the sequence points, without having
> to explicitly code for each case? That would be cool.

You can do much better than that - if the grain of the
processing of the items in the "various stages of request
processing" is appropriate (discrete, probably small) and the
interface to the items is orthogonal, you could dynamically
alter the processing model according to system load or any
other criteria you cared for, by changing the "long list of
sequence points". Now that would be way cool ... and might
also be an area where C++ gives a complexity-reduction
advantage over C.

Regards,

-- 
Mike Anderson
mka@redes.int.com.mx
+52 473 23730 voice/fax
Guanajuato, GTO, Mexico

"If it looks like a bug, waddles like a bug,
and quacks like a bug, its a quack!"