Re: [apache-plusplus] Re: Process model for C++ Apache

[bret <mersault@busboy.sped.ukans.edu>]

	Michael, I agree, we're approaching novel length here.  I'm not in
a position where I can do this in great detail, but I thought I might
offer a brief explanation.  Note I'm doing so without studying the rest of
this thread... I noticed there were a few other messages waiting, and I'm
curious to see what they contain... :)

> Therefore, I believe that that the KCS (dis)advantage
> of start-to-end vs. multiple-thread is roughly a wash.
> Multiple threads per message should allow equal message
> processing performance with a large reduction in thread
> count, which has a performance advantage on most hardware.
> However, start-to-end is easier to conceptualize and
> program.

	Your statements above appear to use kernel context switches as the
sole benchmark in determining the performance of two different
architectures.  Is this the only benchmark we can use to determine thread
performance?  Surely, there is more involved in exerting a condition
variable which does not require a context switch.  If nothing else, it
would seem that there would be _some_ time spent determining which of the
waiting threads gets the mutex.  Surely this time spent in doing things
_other_ than context switching adds up over several steps...?
	I can't really argue this point too well because my knowledge is
so limited... various sources I checked suggested this analysis is
correct, but I should probably shut up before I say something that is just
plain wrong.  Dean, Ben... am I even remotely close with this analysis?
	I have never claimed to be an expert in multithreading.  But for
my money, the simplicity in design and programming would seem
justification enough... :)

	I'll try to limit the novel length by taking out the stuff that
isn't really terribly significant... if I leave out something important,
let me know...

> > >Thread pools are a
> > >natural extension of the more-important abstraction of
> > >connection- and message-pools and the way they abstract
> > >connections from network I/O, and messages from the
> > >processing of messages. A connection is nothing more than
> > >a record in the connection queue; a message is nothing more
> > >than a record in the message-queuing system.
> 
> Bret wrote:
> > I agree with your abstraction regarding a connection and/or
> > message queueing system.  That's fine... but how does a
> > thread pool model become a "natural extension" of that? If
> > by "thread pools" you mean your model of thread pools (i.e.
> > pools of threads performing a single function), I would argue
> > there is at least one other model which is just as natural...
> > a single thread handling a single request over it's entire
> > lifetime.

[using the second wording of your question]
> Let me ask this a different way: of what use is
> the connection- and message-abstractions of pools to the
> start-to-end thread model? If the same thread takes
> connection/message from acceptance through writing back the
> response, there is no significant difference (abstraction)
> between a connection and a message, and a pool is useless
> overhead. Since the connection/message mapping to threads is
> one-to-one, there also is no significant abstraction between
> them. Connection/message/thread is essentially all the same,
> so lets throw away those pesky connection- and message-
> abstractions.

	Actually, there is still a distinct difference between a
connection and a request (what I believe you are referring to as
"message"), especially with HTTP 1.1, where we can have multiple
"requests" for a given "connection".  Pipelining made the difference
between these two explicit... so we still must recognize them as two
different objects.
	If this were not the case, I _might_ agree with you... but I'm not
sure removing the abstraction would be such a bad thing, _particularly_
if in doing so we could remove overhead as well...

> > >Organizing connections and messages in this form allows us
> > >to manipulate these records independent of their content;
> > >we have meta-connection and meta-message information
> > >available.
> 
> > Okay... we still agree... but independent of your method of
> > processing (i.e. the abstraction of each pool performing a
> > "function" as opposed to a single thread handling an entire
> > request for it's entire lifetime), how is this different from
> > any other Web server on the planet?
> 
> Sorry, I guess I don't understand the question. I think the
> connection- and message-abstractions (pools or queues) which
> imply the thread abstraction (pool) is different from the
> integrated connection/message/thread model used by some Web
> servers. Most Web servers already have some sort of pool of
> threads or processes, but with a different abstraction concept,
> and there are certainly other ways of assigning thread
> functionality thats neither start-to-end or what I proposed.

	What I guess I was asking was "With the current Apache, which is
not multithreaded, I can obtain meta-connection and meta-request
information.  The abstractions you outline do not give us anything we
couldn't get other ways.  And _if_ they do introduce overhead without any
significant benefits, why do it?"

> > You've now got(at least) three functions, with three condition
> > variables, along with three mutexes. You can do the same thing
> > with one mutex, wrapped around your accept() call... why
> > _introduce_ complexity if it doesn't buy you anything?
> 
> If the mutexes and CVs are light-weight (no KCS), who cares? It
> is more complex, so we encapsulate the complexity nicely in
> classes.

	It can be easily encapsulated, I agree.  But as I said above, I'm
not convinced that you won't see a performance hit from so many context
switches... I'm not gonna say anything else about that until I hear from
somebody who knows more 'bout threads than me!

> I'm really curious to hear how you are going to handle example
> 2 with a single start-to-end thread; you have multiple requests,
> lets say 3, and a single start-to-end thread. Just to start the
> ball rolling, are you going to:

	Whoa!!!  Who said anything about a single thread?  I'm proposing a
multithreaded Web server, each thread of which follows a single connection
(I changed this after thinking about pipelining) for it's lifetime.
Wrapping a mutex around the accept() call solves the problem nicely...
where's the difficulty?
	My original thought was a single reader, which would toss requests
(that is what you mean by "message", right?) into a queue which would then
get serviced by multiple processing threads.  This requires some of the
abstractions you mention... but I've since revised my design to consist of
multiple start-to-end fibers...

> Bret wrote:
> > I can do the same thing with some type of Socket object
> > that doesn't require the introduction of a new thread pool,
> > again with all the overhead that that brings.  I understand
> > that the generic nature of your thread pool is nice, but if
> > I can do the same thing with objects _without_ having to
> > introduce a new thread every time I want to write somewhere,
> > why not do that?
> 
> What is the new thread pool you are talking about? We have a
> pool of generic threads which are pre-started (although the
> pool expands and shrinks according to load) to minimize
> thread start-up overhead. These threads service the front- and
> back-end (in this example) according to their function. They
> make full use of the connection- and message-abstractions for
> fully asynchronous operation. I guess I don't understand the
> question. How are you going to manage your "pool" of objects?

	"New thread" was poorly phrased... guess I should've said
something like "without _signalling_ a new thread...
	One thing I was kinda curious about in your discussion... you
mentioned that the readers and writers were fully abstracted.  Does that
mean they simply read and write raw data to and from sockets?  What do you
use then to process HTTP headers?  I had assumed this was done by your
readers and writers... which would make them pretty non-general (i.e.
communicating with databases would be non-trivial).  So is this done in
the processors?  If so, do you then have a different processor for each
type of communication you need to do?  Do you need a pool of database
processors, a pool of HTTP processors, etc.?  Just curious...

> > >3. continued: However, since we have a message queue, all
> > >three queries/responses are done simultaneously so total
> > >request processing is 2 seconds. This parallelism is made
> > >possible by the abstraction of messages from message processing
> > >and explains why we have pools of readers(), writers(), etc.
> > >That pool of readers() is working one browser request and
> > >3 back-end DB responses which result from the browser request.
> 
> Bret wrote:
> > I agree with you when you say "The DB processing is probably
> > the bottleneck in this setup..."
> > In the case of multiple databases, maybe.  But in the case of
> > multiple queries to a _single_ database, this just doesn't work.
> > Your numbers assume that the database can handle the concurrency
> > you're looking for. Let's assume the database only receives
> > connections on a single, well-known port... if it get's three
> > requests simultaneously, it will most likely queue the requests
> > and process them one at a time.  At least that's my suspicion.
> > So you're still waiting.  Only now you've got blocked threads
> > sitting around twiddling their thumbs while you're waiting for
> > a database to respond to your query.
> > If that is the case, what has the concurrency in your program
> > bought you but excess overhead?
> 
> Hold-on here! I presented an example which is not uncommon in
> large sites (I've worked on one even bigger than the example)
> and showed how intra-request parallelism (as different from the
> extra-request parallelism of example 2) is possible with the
> connection-, message-, and threading abstractions that I
> suggested.
> 
> It is not a valid response to rewrite the example so that no
> parallelism is possible and then ask "what has the concurrency
> in your program bought you".
> 
> And just to clarify a point in the rewritten example, since these
> generic threads, and the processor() thread are asynchronous,
> there are no "blocked threads sitting around twiddling their thumbs".
> These threads are processing other requests while the DB is doing
> its thing. This is how fewer specialized threads can process more
> requests than the number of threads. However, the start-to-end
> threads will be "twiddling their thumbs" in either the original
> or the rewritten example.

	It _is_ a valid objection to say that your claim of gaining
concurrency applies only in select situations.  _That_ was the point I was
trying to make... in certain situations, it may gain you something, but
how much does this help you generally?
	There are also plenty of sights which perform multiple queries of
the same database... a case where your concurrency will not improve
performance.
	Yes, you're readers would be off servicing other requests.  So
would various other of my start-to-end threads.  I'm assuming you must be
doing processing of DB requests somewhere other than your reader and
writer functions, or else they would be _completely_ non-generalized.  So
_something_, whatever is processing your DB request, is waiting on a reply
and _is_ twiddling it's thumbs".  I'm not sure what the "something" is in
your model (is it a reader?  A processor?  A DB-specific processor?), but
_something_ must be waiting to do something with the data returned by your
queries...

> > On a side note, I'm not sure how well a design such as this lends
> > itself to modularization... you're doing _everything_ within
> > the server core.  I would say you would be better off coding
> > that functionality into an external module, perhaps giving the
> > module an interface to the "thread pools", and letting them
> > perform the query.  The module can be implemented however you'd
> > like... but I guess I just keep wondering why I would use a
> > thread pool model if I can achieve the exact same effect with
> > multiple threads of Socket objects...
> 
> That is an implementation issue. The same functionality can be
> provided very modular or very intergrated according to the
> requirements of the program - in this case an httpd.
> By the way, your "multiple threads of Socket objects" sound
> suspiciously like my thread pool, with similar management
> overhead.

	The whole point I was trying to make with the Socket object is
that it is not necessary to abstract thread pools into specific functions
to obtain some of the benefits that you're speaking of.  In fact, it quite
possibly would bring about better performance to use a different design.
I just don't know enough to be sure...

> > >4. Take the same setup as in example three, but add 3 more
> > >redundant DB servers. Since we can derive meta-connection
> > >information from the connection queue, we can tell if a
> > >DB server is bogging-down and load-balance to the redundant
> > >server.
> 
> Bret wrote:
> > Agreed... I don't think anyone is challenging your connection
> > abstraction or your request abstraction... Apache makes hefty
> > use of something similar right now. Look through the code and
> > check out request_rec and conn_rec (or is it connection_rec?
> > I never can remember)... these structs either do or easily could
> > perform the exact same functions you're talking about. But again,
> > I would argue most of this should be modularized...
> 
> Two points here:
> 1. Certain generic functionalities can be part of the core
> because they apply to all messages, but can also be modular
> in the since that its trivial to remove or modify. These are
> not model issues but implementation issues.

	Agreed... the main point I was making above is that the
meta-information you refer to is already available...

> 2. I don't see usefulness of connection- or message-abstraction
> using the start-to-end threading model. Help me out here.

	I don't have the Apache code available, so I will hold off on
answering this until I've had a chance to look... unless one of the group
members would like to help me out! :)

> > >The process model is more
> > >than the threading model - it is also how the data is
> > >organized.
> 
> Bret wrote:
> > This distinction is crucial... I accept the second part
> > ("how the data is organized"), but I'm highly suspicious of
> > your threading model (actually, in the terms you use below,
> > I believe I should say "process model").  And, apparently like
> > you, I do not view the two as being impossibly intertwined...
> 
> Well, we agree on something.

	We agree on quite a few things... I appreciate your patience,
Michael.  I'm trying not to be overly insistent or anything, so if I sound
belligerant, please understand that I'm just being stubborn... :)

	I'm hoping someone who knows more about threads can offer some
insight about the current performance debate... until then, or until I
find a better source, I'll keep quiet... :)

							- Bret -