Most Frequently Asked Questions

  This is a shorten list of the *most* common questions which have come up
  on the newsgroup. (Somewhat edited by yours truly.)  If you have other
  comments/corrects, just drop me a line (Bil LambdaCS.com). 

-Bil


     ==================================================================
      M O S T   F R E Q U E N T L Y    A S K E D    Q U E S T I O N S 
     ==================================================================

Q3:    What kinds of issues am I faced with in async cancellation?
Q12:  How many threads are too many in one heavyweight process?
Q16:  After 1800 calls to thr_create() the system freezes. ??
Q17:  Compiling libraries which might be used in threaded or unthreaded apps?
Q18:  What's the difference of signal handling for process and thread? 
Q20:  What about using sigwaitinfo()?
Q34:  Sometimes the specified sleep time is SMALLER than what I want.
Q37:  Any Pthreads for Linux?
Q38:  Any really basic C code example(s) and get us newbies started?
Q48:  Are there Pthreads on NT?
Q63:  What's a good way of writing threaded C++ classes?
Q81:  How many threads CAN a POSIX process have? 
Q86:  Is there anything similar to posix conditions variables in Win32 API ?
Q92:  How do priority levels work?
Q93:  C++ member function as the startup routine for pthread_create(). 
Q94:  Spurious wakeups, absolute time, and pthread_cond_timedwait()
Q98:  beginthread() vs. endthread() vs. CreateThread? (Win32)
Q102: Signals and threads are not suited to work together?
Q104: Windows NT Fibers?
Q107: What parts, if any, of the STL are thread-safe?
Q120: Calling fork() from a thread
Q121: Behavior of [pthread_yield()] sched_yield()
Q126: Cancellation and condition variables
Q128: How do I measure thread timings? 
Q129: Contrasting Win32 and POSIX thread designs
Q143: Pthreads and Linux 
Q153: C++ Exceptions in Multi-threaded Solaris Process 
Q155: "lock-free synchronization" 
Q156: Changing single bytes without a mutex 
Q158: VOLATILE instead of mutexes? 
Q167: Using cancellation is *very* difficult to do right...
Q210: Where is the threads standard of POSIX ????
Q211: Is Solaris' unbound thread model braindamaged?
Q212: Releasing a mutex locked (owned) by another thread.
Q293: Details on MT_hot malloc()?
Q354: Using POSIX threads on mac X and solaris?


================================================================
 Q3: What kinds of issues am I faced with in async cancellation?  


Specifically, NO, it is NOT safe to call read() with async cancel. On some
implementations it may work, sometimes. In general, it *MAY* work if, on
the particular release of your particular operation system, read() happens
to be implemented with no user-mode code (aside from a syscall trap). In
most cases, a user mode cancel will NOT be allowed to corrupt kernel data.

However, no implementations make any guarantees about their implementation
of read(). It may be a syscall in one version and be moved partly into
libc in the next version.

Because, as has already been commented, async cancel really isn't very
useful. There is a certain small class of application that can benefit
dramatically from async cancel, for good response to shutdown requests in
long-running compute-bound threads. In a long and tight loop it's not
practical to call pthread_testcancel(). So in cma we provided async cancel
for those cases. In retrospect I believe that's probably one of the bad
parts of cma, which POSIX should have omitted. There may well have been
"hard realtime" people in the room who wanted to use it, though (the POSIX
threads standard was developed by roughly 10 "threads people" and 40 to 50
"realtime people").

------------------------------------------------------------------------
Dave Butenhof                              Digital Equipment Corporation
butenhof@zko.dec.com                       110 Spit Brook Rd, ZKO2-3/Q18
Phone: 603.881.2218, FAX: 603.881.0120     Nashua, NH 03062-2711
                 "Better Living Through Concurrency"
------------------------------------------------------------------------


=============================TOP===================================
 Q12: How many threads are too many in one heavyweight process?    

The answer, of course, is "it depends".

Presumably, the number of threads you're considering far outstrips the
number of processors you have available, so it's not really important
whether you're running on uni- or a multiprocessor, and it's not really
important (in this general case) whether the threads implementation has
any kernel support (presumably it doesn't on HP-UX, judging by your post
from 14 Feb 1996 14:31:42 -0500).  So, it comes down to what these
bazillion threads of yours are actually doing.  

If, for the most part, they just sit there waiting for someone to tickle
the other end of a socket connection, then you can probably create LOTS
before you hit "too many".  In this case it would depend on how much
memory is available to your process, in which to keep all of these
sleeping threads (and how much kernel resources are available to create
sockets for them ;-).

If, on the other hand, every one of these bazillion threads is hammering
away on the processor (trying to compute some fractal or something :-),
then creating any more threads than you have processors is too many.
That is, you waste time (performance, throughput, etc.) in switching
back and forth between the threads which you could be spending on
something useful.  That is, life would be better if you just created a
couple of threads and had them make their way through all the work at
hand.

Presumably, your application falls somewhere between the two extremes.
The idea is to design so that your "typical operating conditions"
involve a relatively small number of threads active at any one time.
Having extra ones running isn't a catastrophe, it just means that things
aren't quite as efficient as they otherwise might be.

-- 

------------------------------------------------------------------------
Webb Scales                                Digital Equipment Corporation
scales@wtfn.enet.dec.com                   110 Spit Brook Rd, ZKO2-3/Q18
Voice: 603.881.2196, FAX: 603.881.0120     Nashua, NH 03062-2711
         Rule #12:  Be joyful -- seek the joy of being alive.
------------------------------------------------------------------------


=============================TOP===================================
 Q16: After 1800 calls to thr_create() the system freezes. ??  

   The default for threads in both UI and POSIX is for threads to be
   "undetached" -- meaning that they MUST be joined (thr_join()).  Otherwise
   their structures will not be freed.  (This default is the wrong choice.  Oh
   well.)

-Bil
=============================TOP===================================
 Q17: Compiling libraries which might be used in threaded or unthreaded apps?  


>   What *is* the straight scoop on how to compile libraries which 
>   might be used in threaded or unthreaded apps?  Hopefully the 
>   "errno" and "putc()" macros will continue to work even if
>   libthread isn't pulled in, so that vendors can make a single
>   version of any particular library.

   Always compile *all* libraries with the reentrancy flag (_REENTRANT for
   UI threads, _POSIX_C_SOURCE=199506L for POSIX threads). Otherwise some 
   poor soul will try to use your library and get hammered.  putc() and
   getc() WILL be slower, but you may use putc_unlocked() & getc_unlocked()
   if you know the I/O stream will be used safely.

   All Solaris libraries are compiled like this.


=============================TOP===================================
 Q18: What's the difference of signal handling for process and thread?   

>   What's the difference of signal handling for process and thread? Do the
>   signals divided into the types of process-based and thread-based which were
>   treated differently in HP-RT? Is there any examples? I'd like to know how to
>   initiate, mask, block, wait, catch, ...... the signals. How can I set the
>   notification list (process or thread?) of SIGIO for both socket and tty
>   using fcntl or ioctl? 

   You probably want to buy one of the books that discuss this in detail.
   Here's the short answer:

	Signal masking is on per-thread based.
	But the signal handlers are per-process based.
	The synchronous signals like SIGSEGV, SIGILL etc will be 
	processed by the thread which caused the signal.

	The other signals will be handled by any ready thread which
	has the mask enabled for the signal.
	
	There are no special thread library for signal handling.


=============================TOP===================================
 Q20: What about using sigwaitinfo()?  

>Here is what I am doing.  I am using the early access POSIX threads.
>My main program blocks SIGUSR1 and creates a number of threads.
>One of these threads is dedicated to this signal.  All it does is a
>sigwaitinfo on this signal, sets a flag when it returns, and exits.
>If I send the SIGUSR1 signal to the process using the kill command
>from another window, it does not seem to get it and the other threads
>(which are doing a calculation in a loop) report that SIGUSR1 is not
>pending.
>
>An earlier version of the program which used a signal handler to set
>the flag worked perfectly.
>
>Do you have any ideas on this?


I assume you are using sigwaitinfo(3r) from libposix4.
Unfortunately, sigwaitinfo() is not MT-safe, i.e. does not work
correctly in an MT program, on 2.3/2.4. Use sigwait(2) - it should
work on 2.3/2.4.  On 2.5 beta, sigwaitinfo() works.

If you really need the siginfo on 2.3/2.4, it is going to be hard,
and the solution depends on whether you are running 2.3/2.4 but here
is an alternative suggestion:

Programmers have used signals between processes as an IPC
mechanism. Sounds like you are trying to do the same. If this is the
case, I would strongly suggest that you use shared memory (see
mmap(2)) between processes and shared memory synchronization (using
the SysV shared semaphores - see semop(2)), or POSIX synchronization
objects with the PTHREAD_PROCESS_SHARED attribute. 

For example, you can set-up a region of shared memory protected by a
mutex and condition variable. The mutex and condition variable would
also be allocated from the shared memory and would be initialized
with the PTHREAD_PROCESS_SHARED attribute. Now, processes which share
this memory can use the mutex and condition variable as IPC
mechanisms - any information that needs to be passed between them can
be passed through the shared memory (alternative to siginfo :-)).

To make this asynchronous, you can have a thread dedicated to
monitoring the shared memory area by waiting on the condition
variable. Now, whenever the signalling process wants to send a
signal, it instead issues a cond_signal on the condition variable.
The thread sleeping on this in the other (receiving) process wakes up
now and processes the information.

In general, signal handlers and threads, even though the system might
support this correctly, should not be used together. Signal handlers
could be looked upon as "substitute threads" when threads were not
around in UNIX, and now that they are, the interactions between them
can be complicated.  You should mix them together only if absolutely
necessary.
 

=============================TOP===================================
 Q34: Sometimes the specified sleep time is SMALLER than what I want.  

>I have a program that generates UDP datagrams at regular intervals.
>It uses real time scheduling for improved accuracy.
>(The code I used is from the Solaris realtime manual.)
>
>This helps, but once in a while I do not get the delay I wanted.
>The specified sleep time is SMALLER (i.e. faster) than what I want.
>
>I use the following procedure for microsecond delays
>
>void
>delay(int us) /* delay in microseconds */
>{
>	struct timeval tv;
>
>	tv.tv_sec = us / 1000000;
>	tv.tv_usec = us % 1000000;
>	(void)select( 0, (fd_set *)NULL, (fd_set *)NULL, (fd_set *)NULL, &tv );
>
>}
>
>
>As I said, when I select a delay, occasionally I get a much smaller delay.
>
>examples:
>	Wanted: 19,776 microseconds, got: 10,379 microseconds
>	Wanted:    910 microseconds, got:    183 microseconds
>
>
>As you can see, the error is significant when it happens.
>It does not happen often. (0.5% of the time)
>
>I could use the usleep() function, but that's in the UCB library.
>Anyone have any advice?

First of all, you can not do a sleep implementation in any increments
other than 10 milliseconds (or 1/HZ variable).

Second, there is a bug in the scheduler (fixed in 2.5) that may
mess up your scheduling in about 1 schedules around every
300,000 or so. 

Third, A much better timing interface will be available in
Solaris 2.6 (or maybe  earlier) thru posix interfaces. That
should give you microsecond resolution with less than 
50 microseconds latency.

Sinan

=============================TOP===================================
 Q37: Any Pthreads for Linux?  

See: http://pauillac.inria.fr/~xleroy/linuxthreads/
http://sunsite.unc.edu/pub/Linux/docs/faqs/Threads-FAQ/html

Linux has kernel-level threads now and has had a thread-safe libc for a
while.  With LinuxThreads, you don't have to worry about things like your
errno, or blocking system calls. The few standard libc functions that are
inherently not thread safe (due to using static data areas) have been
augmented with thread-safe alternatives.

LinuxThreads are not (fully) POSIX, however. 
   
                   -----------------

I'm quite familiar with Xavier's package. He's done an awesome job given
what he had to work with. Unfortunately, the holes are large, and his
valiant attempts to plug them result in a larger and more complicated
user-mode library than should be necessary, without being able to
completely resolve the problems.

Linux uses clone() which is not "kernel-level threads", though, with
some proposed (and possibly pending) extensions in a future version of
the kernel, it could become that. Right now, it's just a way to create
independent processes that share some resources. The most critical
missing component is the ability to create multiple executable entities
(threads) that share a single PID, thereby making those entities threads
rather than processes.

Linuxthreads, despite using the "pthread_" prefix, is NOT "POSIX
threads" (aka "pthreads") because of the aforementioned substantial and
severe shortcoming of the current implementation based on clone().
Without kernel extensions, a clone()-based thread package on Linux
cannot come close to conforming to the POSIX specification. The common
characterization of Linuxthreads as "POSIX threads" is incorrect and
misleading. This most definitely is not "a true pthreads
implementation", merely a nonstandard thread package that uses the
"pthread" prefix.

Note, I'm not saying that's necessarily bad. It supports much of the
interface, and unlike user-mode implementations (which also tend to be
far more buggy than Linuxthreads), allows the use of multiple
processors.  Linuxthreads is quite useful despite its substantial
deficiencies, and many reasonable programs can be created and ported
using it. But it's still not POSIX.

=============================TOP===================================
 Q38: Any really basic C code example(s) and get us newbies started?  

>Could one of you threads gods please post some really, really basic C code
>example(s) and get us newbies started?  There just doesn't seem to be any other
>way for us to learn how to program using threads.
 

The following is a compilation of all the generous help that was posted or mailed to me 
concerning the use of threads in introductory programs.  I apologize for it not being
edited very well...  (Now I just need time to go through all of these)

Here's all of the URL's:
[All of the other previously listed URLs died :-( -Bil]
http://www.LambdaCS.com/classes/classes.html

--Carroll

=============================TOP===================================
 Q48: Are there Pthreads on Win32?  
 
   Yes, there is a GNU pthreads library for Win32.  It is still under
   active development (1999), but you can find out more by looking at
   http://sourceware.cygnus.com/pthreads-win32/

   (This is a combination of Ben Elliston & John Bossom's work. & others?)

=============================TOP===================================
 Q63: What's a good way of writing threaded C++ classes?  

> 
> Ian Emmons wrote:
> >
> > Baard Bugge wrote:
> > >
> > > >How would we put the whole object into a thread?
> > >
> > > Been there. Done that. Let the constructor create a thread before
> > > returning to the caller (another object). But beware, your OS will
> > > propably start the thread by calling a function (specified by you)
> > > C-style. You want this function to be a member function in your class,
> > > which is ok as long as you make it static. The thread function will
> > > also need the this-pointer to your newly created object. What you want
> > > will look something like this (in NT):
> > >
> > > // Thread callback function.
> > > // NOTE: Need to be written in C or be a static member function
> > > // because of C style calling convention (no hidden this pointer)
> > > LPTHREAD_START_ROUTINE CThread::ThreadFunc(LPVOID inputparam)
> > > {
> > >    CThread *pseudo_this = (CThread *) inputparam;
> > >    ...
> > > }
> > >
> > > This function have access to all the members in the object through the
> > > pseudo this pointer. And all member functions called by this function
> > > will run in the same thread. You'll have to figure out how to
> > > communicate with the other objects in your system though. Be careful.
> > >
> > > --
> > > BaBu
> >
> > You can take this even a step further.  Add a pure virtual to your generic
> > CThread class like so:
> >
> > class CThread
> > {
> >       ...
> > protected:
> >     // I don't remember what Win32 expects as the return value, here,
> >     // but you can fix this up as you wish:
> >     virtual unsigned entryPoint() = 0;
> >       ...
> > };
> >
> > Then have the static ThreadFunc call it like so:
> >
> > // Thread callback function.
> > // NOTE: Need to be written in C or be a static member function
> > // because of C style calling convention (no hidden this pointer)
> > LPTHREAD_START_ROUTINE CThread::ThreadFunc(LPVOID inputparam)
> > {
> >    return ((CThread*) inputparam)->entryPoint();
> > }
> >
> > Now, to create a specific thread, derive from CThread, override entryPoint,
> > and you no longer have to mess around with a pseudo-this pointer, because
> > the real this pointer is available.
> >
> > One tricky issue:  make sure you differentiate between methods that the
> > thread itself will call, and methods that other threads (such as the one
> > that created the thread object) will call -- you will need to do thread
> > synchronization on class members that are shared data.
> >
> > Ian
> >
> > ___________________________________________________________________________
> > Ian Emmons                                       Work phone: (415) 372-3623
> > ian@persistence.com                              Work fax:   (415) 341-8432
> > Persistence Software, 1720 S. Amphlett Blvd. Suite 300, San Mateo, CA 94402
> 

------------------
OK, let me warn everyone this is a very long response, but I just came off
of a large radar project on which I had to design multithreaded objects so
this question jumped out at me.

Yousuf Khan  wrote in article
<32CD2EAB.5678@bellglobal.com>...
> I got some hypothetical questions here, I'm not actually now trying to
> do any of this, but I can see myself attempting something in the near
> future.
> 
> Okay, I'm thinking multithreading and OO design methodologies are
> tailor-made for each other, in theory at least. OO design mandates that
> all object instances are considered concurrent with each other. That
> seems like a perfect application of threading principles. However,
> current threading protocols (such the POSIX Pthreads, Microsoft/IBM
> threads, Sun UI threads, etc.) seem to be based around getting
> subprocedures threaded, rather than getting objects threaded.

First, let me state my own programming background so you can apply the
appropriate grain of salt to what I say and understand my assumptions.  I
have programmed first for a few years in a DEC, VMS environment and then for
several more in a Windows/Windows NT environment.

> Okay, I suppose we can get individual methods within an object to be
> threaded, because they are just like subprocedures anyways. But what if we
> wanted to be extremely pedantic, and we want the entire object to be in
> its own thread, in order to be true to OO design paradigms? How would we
> put the whole object into a thread?  My feeling is that we should just
> call the object's constructor inside a thread wrapper, that way the entire
> object will go into a thread, including any other methods that are part of
> that object. What I guess I'm saying is that will calling the constructor
> inside a thread wrapper, only run the constructor inside that thread and
> then the thread will end, or will the entire object now run inside that
> thread from now on?  Am I being oversimplistic in my speculation?


If you want to force an object to, as you say, "run in one thread", you
would have to be able to make public every member function perform a context
switch to the desired thread upon entering the function and switch back upon
exiting.  You would have to protect all member variables and use Get/Set
functions for them that performed context switches as well.

Under Windows NT, if you send a message to a window created by a different
thread, that context switch is performed for you by the operating system.
Your process waits until the ::SendMessage() call completes.  Other than
using SendMessage(), I do not know how you would accomplish such an
operation.  And SendMessage requires a window to which the message will be
sent.  Thus, under NT, you would have to make your object create some kind
of hidden window in the context of the desired thread and then have every
member function do a ::SendMessage() to that window.

(There are variations -- e.g. SendMessageCallback(), PostMessage(), etc for
asynchronous function calls)

Such a design is possible, and maybe workable, but seems to defeat the
purpose of threads, doesn't it?  If one thread is just going to have to wait
for the special thread every function call, why have the special thread at
all?

And I haven't even considered OLE and accessing objects across process
boundaries, or thread-local storage.

(Again, I'm speaking pretty exclusively about the NT threading model here.
I've had enough VMS to last me a lifetime and know very little about Posix
threads.)

It seems your reason for wanting the entire object to run in its own thread
is to be true the OO paradigm, but I think that's perhaps too much of a good
thing.

Why not make your objects completely thread-safe instead?  Create some sort
of a Single-Writer / Multiple-Reader resource locking object for all objects
of the class.  Make each member function use this resource guard, acquiring
a read-lock if it's a const member function or write-lock if it is not
const.

There's nothing to prevent you from assigning specific threads to the
objects to do background work on them, but as long as all access to the
objects is through those safe member functions, they are completely thread
safe..

I mention this because this is how I designed a large radar project I just
finished working on.  I used completely thread-safe, reference counted
objects, read/write locks, and smart pointers in my design and the results
were far better than my most optimistic hopes.  A very fast workstation
program with many dynamic displays showing an enormous amount of continously
changing data stored in a SQL server database.

I've gone on way too long here so I'll end this without saying half of what
I want to say.  Hope this gives you a few ideas.

=============================TOP===================================
 Q81: How many threads CAN a POSIX process have?  

> r> _POSIX_THREAD_THREADS_MAX that claims to be the maximum threads per
> r> process.
>
> > As I recall, this is a minimum requirement.  Solaris certainly
> supports far more than 64 threads in a single process, and I'm sure
> that Irix does, too.

POSIX specifies two compile-time constants, in , for each
runtime limit. One is the MINIMUM value of that MAXIMUM which is
required to conform to the standard. _POSIX_THREAD_THREADS_MAX must be
defined to 64 on all conforming implementations, and all conforming
implementations must not arbitrarily prevent you from creating at least
that many threads.

The symbol PTHREAD_THREADS_MAX may ALSO be defined, to the true limit
allowed by the system, IF (and only if) that limit is fixed and can be
predicted at compile time. (The value of PTHREAD_THREADS_MAX must be at
least 64, of course.) I don't know of any systems that define this
symbol, however, because we don't implement any fixed limit on the
number of threads. The limit is dynamic, and dictated purely by a wide
range of resource constraints within the system. In practice, the only
way to predict how many threads you can create in any particular
situation is to bring a program into precisely that situation and count
how many threads it can create. Remember that the "situation" includes
the total size of your program text and data, any additional dynamic
memory used by the process (including all shared libraries), the virtual
memory and swapfile limits of the current system, and, in some cases,
the state of all other processes on the system.

In short, the concept of "a limit" is a fiction. There's no such thing,
without knowing the complete state of the system -- rarely practical in
real life.

Oh, and by the way, there's no guarantee (in POSIX or anywhere else)
that you can create even 64 threads. That just means that the system
cannot arbitrarily prevent you from creating that many. If you use up
enough virtual memory, you may be unable to create even one thread.
That's life.

As Bryan said, you can normally rely on being able to create hundreds,
and usually thousands, of threads on any of the current 2-level
scheduling POSIX threads implementations. Kernel-level implementations
are typically more limited due to kernel quotas on the number of kernel
thread slots available for the system and often for each user.

=============================TOP===================================
 Q86: Is there anything similar to posix conditions variables in Win32 API ?  

Ian Emmons wrote:
> 
> Dave Butenhof wrote:
> >
> > kumari wrote:
> > >
> > > Is there anything similar to posix conditions variables in Win32 API ?
> > > Thanks in advance for any help.
> > > -Kumari
> >
> > The answer depends very much upon what you mean by the question. Win32
> > has "events", which can be used to accomplish similar things, so the
> > answer is clearly "yes". Win32 events, however, behave, in detail, very
> > differently, and are used differently, so the answer is clearly "no".
> > Which answer do you prefer? ;-)
> 
> Good answer, Dave.  This is one of the most frustrating things about the
> Win32 threading API.  CV's are incredibly powerful and fairly easy to use,
> but Win32 unfortunately ommited them.
> 
> In WinNT 4.0, there is a new API called SignalObjectAndWait which can be used
> to implement a CV pretty easily.  There are two problems:
> 
> (1) This API is not available on WinNT 3.51 or Win95.  Hopefully it will show
> up in Win97, but I don't know for sure.
> 
> (2) Using this API with a mutex and an auto-reset event, you can create a
> CV-lookalike where PulseEvent will behave like pthread_cond_signal, but there
> is no way to immitate pthread_cond_broadcast.  If you use a mutex and a
> manual event, PulseEvent will behave like pthread_cond_broadcast, but there
> is no way to immitate pthread_cond_signal.  (Sigh ...)
> 
> I know ACE has a Win32 CV that works in general, but I seem to recall Doug
> Schmidt saying that it's very complex and not very efficient.
> 
> Ian 

=============================TOP===================================
 Q92: How do priority levels work?  

Kamal Kapila wrote:
> 
> Hi there,
> 
> I'm working on an internal package to provide platform independant
> thread services (the initial platforms are DECUNIX 4.0 and Windows NT).
> The problem I'm having is understanding the thread scheduling on
> DECUNIX.
> 
> It would seem to me logical that the threads of a process would have the
> same priority and policy of their associated process by default.
> However, when I check the process priority/policy I get completely
> different values from when I check the individual thread priorities and
> policies.  In fact, the priority values do not seem to even follow the
> same scale (I can set process priorities from 0-63, while thread
> priorities go only from 0-31).  In addition, setting the process
> priority does not seem to effect the thread priorities at all (!).

Basically, there are "interaction issues" in implementing a 2-level
scheduling model (as in Digital UNIX 4.0), that POSIX didn't attempt to
nail down. We deferred dealing with these issues until some form of
industry concensus emerged. That industry concensus has, since, not
merely "emerged", but has become a mandatory standard in the new Single
UNIX Specification, Version 2 (UNIX98).

With 2-level scheduling, it really doesn't make much sense to "inherit"
scheduling attributes from the process -- because those attributes MEAN
entirely different things. Digital UNIX, by the way, doesn't really have
a "process" -- it has (modified) Mach tasks and threads. (There are a
set of structures layered over tasks to let the traditional UNIX kernel
code deal with "processes" in a more or less familiar way, but a process
is really sheer illusion.) Since tasks aren't scheduled, they really
have no scheduling attributes -- threads do. Since a non-threaded
process has a task and a single thread, the 1003.1b (realtime)
scheduling functions operate, in general, on the "initial thread" of the
specified "process".

The kernel thread scheduling attributes control scheduling between
various kernel threads. But a POSIX thread is really a user object, that
we map onto one or more kernel threads (which we call "virtual
processors"). Pretending to set the scheduling attributes of this thread
to the "process" attributes makes no sense, because the scheduling
domain is different. POSIX threads are scheduled only against other
threads within the process -- not against kernel threads in other
processes.

POSIX provides a way to create threads that you really want to be
scheduled against other kernel threads -- essentially, forcing the POSIX
thread to be "bound" to a kernel thread itself, at the expense of (often
substantially) higher scheduling costs. This is called "system
contention scope". Digital UNIX 4.0 didn't support system contention
scope (which is an optional feature of POSIX), but we've added it for
the next version (4.0D).

Each system contention scope (SCS) thread has its own scheduling
attributes, independent of the process. While it might make some
intuitive sense to inherit the process priority, POSIX doesn't provide
any such semantics. A newly created thread either has explicit
scheduling attributes, or inherits the attributes of the thread that
created it. Of course, since setting the "process" attributes affects
the initial thread, threads that IT creates will inherit the "process"
attributes by default. But changing the "process" attributes won't (and
shouldn't) affect any SCS threads in the process.

The ambiguity (and the only relevant question for the implementation
you're using, which doesn't support SCS threads), is, what happens to
the virtual processors that are used to execute POSIX threads, when the
"process" scheduling attributes are changed? And with what attributes
should they run initially? UNIX98 removes the (intentional) POSIX
ambiguity by saying that setting the 1003.1b scheduling attributes of
the "process" WILL affect all "kernel entities" (our virtual processors,
Sun's LWPs) used to execute process contention scope (PCS, the opposite
of SCS) threads. By extension, the virtual processors should initially
run with the existing process scheduling attributes.

This will be true of any UNIX98 branded system -- but until then,
there's no portable rules.

The fact that the POSIX thread interfaces don't use the same priority
range as the system is a stupid oversight -- I just didn't think about
it when we converted from DCE threads to POSIX threads for 4.0. This has
been fixed for 4.0D, though it's a bit too substantial a change (and
with some potential risk of binary incompatibilities) for a patch.

> (BTW, I am using sched_getparam() and sched_getscheduler() to get the
> process related values and  pthread_getparam() to get the thread related
> values).

Right.

> Specifically, I have the following questions :
> 
> - What is the relationship between the process priority/policy and the
> thread priority and policy  ?

There's very little relationship. Each POSIX thread (SCS or PCS) has its
own scheduling attributes (priority and policy) that are completely
independent of "process" attributes. UNIX98, however, says that the
"kernel entities" used to execute PCS POSIX threads WILL be affected by
changes to the "process" scheduling attributes -- but SCS threads will
not (and should not) be affected by such changes. (Nor will the
scheduling attributes of PCS threads, even though their "system
scheduling attributes" effectively come from the virtual processor,
which is affected.)

> - Does the scheduler schedule individual threads independently, or are
> processes scheduled, with a process's threads then sharing the process
> CPU time?

As I said, there's no such thing as a process, and the closest analog,
the Mach task, isn't a schedulable entity. All threads are scheduled
independently -- each has its own scheduling attributes, its own time
slice quantum, etc. On Digital UNIX 4.0, with only PCS threads, the
kernel schedules the virtual processor threads of all the processes
(plus the single kernel threads associated with all non-threaded
processes). Threaded processes also contain a user-mode scheduler, which
assigns PCS threads to the various virtual processors, based on the PCS
thread scheduling attributes. (A process has one virtual processor for
each available physical processor on the system.)

On Digital UNIX 4.0D, with SCS thread support added, each process may
also have any number of SCS threads, which map directly to individual
and independent kernel threads. SCS threads are scheduled the same as
virtual processors -- each has its own scheduling attributes, time slice
quantum, etc.

(It might seem that managing CPU time by kernel threads rather than by
processes allows users to monopolize the system by creating lots of
kernel threads. But they could do that by creating lots of processes,
too... and a kernel thread is cheaper for the system than a process,
which is really a thread plus a task. The ability to create new kernel
threads, as well as processes, is limited both by user and system
quotas. And of course, in 4.0, users can't actually create new kernel
threads -- only POSIX threads, which are mapped to the process' existing
virtual processors.)

So each process presents a set of runnable kernel threads to the kernel:
A mix of SCS threads and the various PCS threads currently mapped on to
one or more virtual processors. The kernel then determines which kernel
threads to schedule on each processor. (That's why it's called "2-level
scheduling".)

> - Is the thread's overall priority a combination of the process priority
> and the individual thread priority ? If so, how is this determined ?

Currently, "process" priority is irrelevant for a threaded process.
Virtual processors don't inherit the process priority. (Actually, they
sort of do, and the first virtual processor is the initial process
thread, which can be changed using the 1003.1b functions -- but the
kernel generates "replacement" virtual processors at various times, and
these currently are always set to the default scheduling attributes
[timeshare policy and priority 19].)

POSIX thread priority determines which threads the user-mode scheduler
assigns to the various virtual processors. Because the virtual processor
priority doesn't change (the whole point of 2-level scheduling is to
avoid expensive kernel calls), the POSIX thread priority has no effect
on the kernel scheduling. That's OK, except in rare cases where
applications comprising multiple PROCESSES have threads (in different
processes) that really need to directly preempt each other based on
priority.

> I have read through all of the Digital documentation that I have but I
> have not been able to find any clear answers to my questions.

A description of the behavior (though in less technical/internal detail
than the one in this posting) can be found in Appendix A (section A.3)
of the Digital UNIX "Guide to DECthreads" manual.

/---------------------------[ Dave Butenhof ]--------------------------\
| Digital Equipment Corporation                   butenhof@zko.dec.com |
| 110 Spit Brook Rd ZKO2-3/Q18       http://members.aol.com/drbutenhof |
| Nashua NH 03062-2698       http://www.awl.com/cp/butenhof/posix.html |
\-----------------[ Better Living Through Concurrency ]----------------/



=============================TOP===================================
 Q93: C++ member function as the startup routine for pthread_create().  

You are correct.  Most C++ compilers DO treat static member functions
like ordinary C functions, so it's usually possible to pass static C++
member functions as arguments to thread creation functions.  However,
some compilers treat them differently, e.g., the OpenEdition MVS C++
compiler doesn't allow static member functions to be used where
ordinary C functions are used, which is a PAIN.

BTW, if you program with ACE
(http://www.cs.wustl.edu/~schmidt/ACE.html) it hides all of this
madness from you so you can write a single piece of source code
that'll work with most C++ compilers.

Take care,

        Doug
                 -------------------


Do I have to? Oh well here goes....

This still uses a nasty cast.  It is also not a good idea to
start a thread in a constructor for the simple reason that
the thread may run _before_ the object is constructed - this
is even more likely if this a base class - I know, I've been
there and done that.

Use an extern "C" friend as in the following compete example:

#include 
#include 

extern "C" void* startIt( void* );

class Fred
{
  pthread_t tid;
  friend void* startIt( void* );
  void* runMe() throw() { std::cout << "Done" << std::endl; return NULL; }

public:
  int start() throw() { return pthread_create( &tid, NULL, startIt, this ); }
  pthread_t id() const throw() { return tid; }
};

void* startIt( void* p )
{
  Fred* pF = static_cast(p);
  return pF->runMe();
}

int main()
{
  Fred f;
  int  s;
  if( (s = f.start()) )
    return s;
  std::cout << "Started" << std::endl;
  void* status;
  pthread_join( f.id(), &status );
  pthread_exit( 0 );
}

> > Timmy Whelan wrote:
> >
> > > You can also make the member function static:
> > >
> >
> > For the Nth time, static members are _NOT_ the same as extern "C" functions.
> > Thier linkage may be different.  Use a friend defined as extern "C" or make
> > the real start member public.
> >

One minor point: Calling convention is, in the general case, a
compiler-specific thing and not an operating-system-specific thing.  Different
compilers for the same operating system can easily have calling conventions
for functions with "C" or "C++" linkages that are incompatible.  

(Some platforms/operating systems have an ABI standard that defines the C
language calling conventions for the platform and operating system.  This is
not universally the case, however.  It is especially not the case for x86
platforms running non-Unix operating systems.)

=============================TOP===================================
 Q94: Spurious wakeups, absolute time, and pthread_cond_timedwait() 

[Bil: The summary is "Retest conditions with CVs." and "The time-out is an
absolute time because it is."  (NB: Deltas are a proposed extension to POSIX.)
This is a nice exposition.]

Brian Silver wrote:
> 
> Ben Self wrote:
> >
> [Snip]
> > The standard specifies that pthread_cond_wait() and
> > pthread_cond_timedwait() may have spurious wakeups.  The reason for this
> > is that a completly reliable once and only once wake up protocol can be
> > excessively expensive for some asymetric multiprocessor systems.
> 
> Well, maybe I'm being a bit anal about this, but this
> really isn't the case. If it was, then you'd have the
> same issue for mutexes as well, and the standard does
> not allow for spurious wakes on mutexes.

[Bil: Actually, you DO get spurious (define "spurious"!) wakeups with mutexes,
YOU just never see them.]
 
> The "while(predicate)/wait" construct is very common
> in concurrent environments (regardless of their symetry).
> The reason is that since the environment is highly
> unpredictable, while you were coming back out of the
> wait, the state of the thing that you were waiting for
> may have changed.
> 
> This construct is used to impliment mutexes as well,
> its just that you don't see it since the predicate
> is known; it is the state of the mutex lock. Cv's force
> the construct to the user code because the predicate
> is not known to the impliment of the cv itself. The
> warnings about spurious wakes are taken seriously when
> mutexes are implimented, and are accounted for in the
> exact same "while(predicate)/wait" construct.
> 
> Wake-only-once doesn't really help. It will remove the
> addition of spurious wakes, but it won't account for
> "valid wake, but the predicate changed". Implimenting
> wake-only-once is expensive when you consider that this
> solution solves both problems.
> 
> Also note that the mutex lock around the predicate
> doesn't solve this problem either. There is a race
> that starts once you see the wake and ends once you
> reaquire the mutex. In that time, another thread can
> get the mutex and change the data (believe me, it
> happens - more often than you'd expect). When you
> reaquire the mutex, and exit the wait, the predicate
> has changed and you'll need to go back to waiting.
> 
> Now, a wake-only-once,-and-gimme-that-mutex atomic
> operation might be nice .
> 
> Brian.

I am not  reposting to be defensive or argumentative.  Upon reflection,
however, I have come to the conclusion that neither I nor subsequent
posters have really dealt with the original poster's question let alone
the new topics that we have thrown about. 

Since this is a response largely to Brian Silver's post, a person I have
a good deal of respect for, I have chosen to include some quotes form
Dave Butenhof's book, Programming with POSIX Threads, because we both
know it and have a mutual admiration for his work.

First of and most importantly the original question I attempted to
answer was:

Fred A. Kulack wrote:
> A side question for the rest of the group...
> All the applications I've seen use a delta time for the wait and must
> calculate the delta each time a cond_timedwait is done. What's the rational
> for the
> Posix functions using an ABSOLUTE time?

I was hoping for an uncomplicated answer and the spurious wakeup issue
seemed to fit the bill.  My writing and thinking however was too
simplistic to provide any meaningful insight.  So I will try again. 
Please realize that some of what you will read below is purely personal
supposition.  Chime in if I have misinformed.

1)  Spurious wakeup is a reason for passing a absolute value to
cond_timedwait.  It is not the reason or even a particularly important
reason.  The standard (POSIX 1003.1c-1995) specifically states that a
compliant implementation of pthread_cond_timedwait() may suffer from
spurious wakeups.  It therefore is reasonable to use and absolute
timeout value instead of an delta to simplify the act of retry.

2)  More importantly it is also very likely a performance issue.  Most
systems when scheduling a software interrupt use an absolute value that
reflects an offset into the OS's epoch.  To constantly be re-evaluating
a delta in user code is excessively expensive especially if most systems
really want an absolute value anyway.

3)  Also their is the reality that the structure timespec is the high
resolution time value of choice in POSIX.  And timespec happens to
represent its time as absolute time.  Add into that the needs of the
powerful realtime group that had a great impact of the shape of POSIX
1003.1c.   What integral unit would we use for a delta anyway? and would
it be in nanoseconds?  Eeak!

4)  Most importantly one would hope that the interface were constructed
to promote good coding techniques.  As Brian Silver stated the
"while(predicate)/wait" idiom is an important technique for far more
reasons than just spurious wakeups.  By using an absolute timeout value
as opposed to a delta this idiom is directly supported by easing its
use.

When I originally brought up the  "while(predicate)/wait" idiom it was
because spurious wakeups would necessitate retrying the predicate.  I
did not intend to state that this was the only or even a particularly
important reason for the pattern.  The while "while(predicate)/wait"
idiom or an equivalent is essential to programming with condition
variables.

1)  Most importantly is the reason Brian silver stated, "There is a race
that starts once you see the wake and ends once you reacquire the
mutex."  It would be difficult and detrimental to concurrency to
construct through synchronization a situation that did not require
re-testing of the predicate after a wakeup.  This is why Brian's magic
bullet "wake-only-once,-and-gimme-that-mutex atomic" does not exist. 
Although it would be nice.

2)  Spurious wakeups do exist. Be consoled by the fact that "The race
condition that cause spurious wakeups should be considered rare.
[Butenhof]"

3)  Also It enables a powerful technique that I have been using for a
several years with great success that Dave Butenhof refers to as "loose
predicates".  "For a lot of reasons it is often easy and convenient to
use approximations of actual state.  For example, 'there may be work'
instead of 'there is work'."  I will go one step beyond that in my
experience of coding distributed web servers there are situations when
the notification mechanism cannot know with certainly that there is work
without actually have performed the entirety of the task itself.  Often
the best a distributed component has to work with is trends and
potentialities.

Lastly, (whew ;) I believe that I have overstated the significance of
the performance implications of only once wakeups.  "Excessively
expensive" is a bit strong without further qualification.  If it were
such a paramount issue Brian Silver is right, mutexes would suffer from
the same restrictions and they absolutely do not.  

There is a performance issue that I have run across many times and have
seen cited in many references including : "Spurious wakeups may sound
strange but on some multiprocessor systems, making condition wakeup
completely predictable might substantially slow all condition variable
operations. [Butenhof]"  Never-the-less, it is the fact that making
wakeup completely predictable does not get you that much.  You still
need to retest your predicate.  In the end it is such an easy and cheap
thing when taken in the context of the overhead of the synchronization
and latency of the wait.

--ben


-----------
Ben R. Self
bself@opentext.com
 


=============================TOP===================================
 Q98: beginthread() vs. endthread() vs. CreateThread? (Win32) 

[Bil: Look at the description in "Multithreading Applications in Win32" (see
books.html)]

Mark A. Crampton wrote:
> 
> Juanra wrote:
> >
> > I'm a Windows 95 programmer and I'm developing a multithreaded
> > server-side application. I use the CreateThread API to create a new
> > thread whenever a connection request comes. I've read that it's better
> > to use beginthread() and endthread() instead of CreateThread because
> > they initialize the run time libraries. What happens with win32
> > CreateThread function?. Doesn't it work properly?. If not, I can't use
> > beinthread because I can't create my thread in a suspended mode and
> > release it after.
> >
> > Does the function beginthreadNT() work under win95?
> 
> No
> 
> >
> > Thanks in advance.
> > Juan Ra.
> 
> Answer to beginthread - use _beginthreadex, which uses same args as
> CreateThread (you can create suspended).  _beginthreadex works on 95 &
> NT but not Win32S.  The priviledge flags are ignored under 95.
> 
> CreateThread _works_ OK - it just doesn't free memory allocated on the C
> run-time library stack when the thread exists.  So you can attempt to
> clean up the runtime library stack, use _beginthreadex, or not use any C
> run time library calls.


=============================TOP===================================
 Q102: Signals and threads are not suited to work together? 

Keith Smith wrote:
> 
> This is a question I posed to comp.realtime, but noticed that you have a
> discussion going on here....  can you offer me any assistance?
> HEre's the excert:
> 
> Shashi:
> 
> Based on your previous email (below), I have a couple of questions:
> 
> 1. If signals and threads are not suited to work together, what
> mechanism can/should be used to implement timing within a thread.  If I
> have two threads that performed autonomous time-based functions, I want
> to be able to have a per-thread timing mechanism.
> 
> 2. If the approach of "block all signals on all threads and send various
> signals to a process - putting the emphasis on the thread to unblock the
> appropriate signals", how do we deal with other threads which may be
> interrupted by a blocked signal (e.g. a read() call that returns EINTR
> even when its thread blocks the offending signal.  Isn't this a flaw?
> This requires the need for a signal handler (wastefull) with the RESTART
> option speicified.
> 
> It seems like a per-thread mechanisms is needed... how does NT
> accomplish this?
> 
> ** I know I shouldn't be relying on a per-LWP signal, but how else can I
> accomplish what I am trying to do?
> 
> In message " timer's interrupting system calls... HELP", Shashi@aol.com
> writes:
> 
> >Hi,
> >Signals and threads are not suited to work together. Personally, I feel that
> >UNIX has a serious flaw in the sense that most blocking systems calls (e.g.
> >read, semop, msgrcv etc) do not take a timeout parameter. This forces
> >programmers to use alarms and signals to interrupt system calls. I have
> >worked with operating systems such as Mach, NT and other which do not suffer
> >from this problem. This makes porting single threaded applications from UNIX
> >(which rely on signals) to a multithreaded process architecture difficult.
> >
> >Even though I come from an UNIX background (Bell Labs in late 80's) I have
> >learnt the hard way that signals make program much more error prone. I have
> >worked extensively on Mach and NT and never saw a reason to use threads. As
> >far as POSIX.1c is concerned I think they did a favor to the users of threads
> >on UNIX by mandating that signals be a per-process resource. You have to
> >understand that LWP is more of a System R4 concept (same on Solaris) and not
> >a POSIX concept. Two level scheduling is not common on UNIX systems (those
> >who implement have yet to show a clear advantage of two level scheduling).
> >
> >I am sure that Dave Butenhof (frequent visitor to this newsgroup) would have
> >more insight as to why POSIX did not choose to implement signals on a
> >per-thread basis (or LWP as you say). I would advice that you should
> >rearchitect your application not to depend on per-thread (LWP) signals. I
> >feel you will be better off in the long run. Take care.
> >
> >Sincerely,
> >Shashi
> >


=============================TOP===================================
 Q104: Windows NT Fibers? 

Ramesh Shankar wrote:
> 
> Found some info. on Windows NT Fibers in "Advanced Windows." Just
> wanted to verify whether my understanding is correct.
> 
> - Is a (primitive) "many to one" thread scheduling model.
> - Fibre corresponds to Solaris "threads" (NT threads then correspond
> to Solaris LWP).
> - If a fibre blocks, the whole thread (LWP for us) blocks.
> - Not as sophisticated as Solaris threads.

  Kinda-sorta.  Certainly close enough.  My understanding is that fibers
were built especially for a couple of big clients and then snuck their way 
out.  As such, I would avoid using them like the plague.  I've read the
APIs and they scare me.

-Bil
			------------------------

Jeffrey Richter, Advanced Windows, 3rd Ed., p.971 states that

"The fiber functions were added to the Win32 API to help companies
quickly port their existing UNIX server applications to Windows NT."

(blah)

The following sentences say that fibers are targeted to the
proprietary user level thread-like quirks some companies did for
whatever reason (ease of programming, performance).

To answer your question: fibers are not an integral part of any MS
application, and I can't imagine that they use it internally anywhere,
and thus won't achive the stability. Does this argument weigh a bit
against their use in a new program?

Joerg

PS: Have you noticed that I managed to keep from flaming :-)


=============================TOP===================================
 Q107: What parts, if any, of the STL are thread-safe? 


Matt Austern wrote:
> 
> Boris Goldberg  writes:
> 
> > > >I'm finding a memory leak in the string deallocate() (on the call to
> > > >impl_->deallocate()) under heavy thread load, and it brings up a
> > > >frightening question:
> > >
> > > >What parts, if any, of the STL are thread-safe?
> > >
> > 
> > STL thread safety is implementation-dependent. Check with
> > your vendor. Many implementations are not thread-safe.
> 
> One other important point: "thread safety" means different things to
> different people.  Programming with threads always involves some
> cooperation between the language/library and the programmer; the
> crucial queston is exactly what the programmer has to do in order to
> get well-defined behavior.
> 
> See http://www.sgi.com/tech/stl/ for an
> example of an STL thread-safety policy.  It's not the only conceivable
> threading policy, but, as the document says, it is "what we believe to
> be the most useful form of thread-safety."
 

                 ---------------------------


I thought all those concerned with developing multi-threaded software
using the STL and C++ might be interested in the topic of STL and thread
safety.  I just bought the July/August 1998 issue of C++ Report and
within there is an article concerning the testing for thread safety of
various popular implementations of STL.  These are the published
results:

 STL implementation     Thread-safe?
 ------------------     ------------
 Hewlett-Packard            NO
 Rogue Wave                 NO
 Microsoft                  NO
 Silicon Graphics           YES
 ObjectSpace                YES

========================  But:   ================

You've missed rather a lot of discussion on this topic in the intervening
months. A few supplemental facts:

1) The definition of ``thread safe'' while not unreasonable is also not
universal. It is the working definition promulgated by SGI and --surprise! --
they meet their own design requirement.

2) Hewlett-Packard provided the original STL years ago, at a time when
Topic A was finding a C++ compiler with adequate support for templates.
Thread safety was hardly a major design consideration.

3) Rogue Wave was quick to point out that the version of their code
actually tested was missing a bug fix that had been released earler.
The corrected code passes the tests in the article.

4) Microsoft has been the unfortunate victim of some messy litigation.
(See web link in footer of this message.) If you apply the fixes from:

http://www.dinkumware.com/vc_fixes.html

then VC++ also passes the tests in the article. Its performance also
improves dramatically.

The C++ Report ain't Consumer Reports. Before you buy on the basis
of an oversimplified table:

a) Make sure your definition of ``thread safety'' agrees with what the
vendor provides.

b) Make sure you get the latest version of the code.

P.J. Plauger
Dinkumware, Ltd.
http://www.dinkumware.com/hot_news.html

=============================TOP===================================
 Q120: Calling fork() from a thread 

> Can I fork from within a thread ?

Absolutely.

> If that is not explicitly forbidden, then what happens to the other threads in
> the child process ?

There ARE no other threads in the child process. Just the one that
forked. If your application/library has background threads that need to
exist in a forked child, then you should set up an "atfork" child
handler (by calling pthread_atfork) to recreate them. And if you use
mutexes, and want your application/library to be "fork safe" at all, you
also need to supply an atfork handler set to pre-lock all your mutexes
in the parent, then release them in the parent and child handlers.
Otherwise, ANOTHER thread might have a mutex locked when one thread
forks -- and because the owning thread doesn't exist in the child, the
mutex could never be released. (And, worse, whatever data is protected
by the mutex is in an unknown and inconsistent state.)

One draft of the POSIX standard had included the UI thread notion of
"forkall", where all threads were replicated in the child process. Some
consider this model preferable. Unfortunately, there are a lot of
problems with that, too, and they're harder to manage, because there's
no reasonable way for the threads to know that they've been cloned. (UI
threads allows that blocking kernel functions MAY fail with EINTR in the
child... but that's not a very good basis for a recovery mechanism.)
After much discussion and gnashing of teeth and tearing of hair, the
following draft removed the option of forkall.

> Is there a restriction saying that it's OK provided the child immediately does
> an exec ?

Actually, this is the ONLY way it's really safe, unless every "facility"
in the process has proper and correct forkall handling to protect all of
the process state across the fork.

In fact, despite the addition of forkall handlers in POSIX 1003.1c, the
standard specifically says that the child process is allowed to call
only async signal safe functions prior to exec. So, while the only real
purpose of forkall is to protect the user-mode state of the process,
you're really not guaranteed that you can make any use of that state in
the child.

> What if I do this on a multiprocessor machine ?

No real difference. You're more likely to have "stranded" mutexes and
predicates, of course, in a non-fork-safe process that forks, becuase
other threads were doing things simultaneously. But given timeslicing
and preemption and other factors, you can have "other threads" with
locked mutexes and inconsistent predicates even on a uniprocessor.

Just remember, that, in a threaded process, it's not polite to say "fork
you" ;-)

/---------------------------[ Dave Butenhof ]--------------------------\
> David Butenhof wrote:
>> 
>> The "UI thread" version of fork() copies ALL threads in the child. The
>> more standard and reasonable POSIX version creates a child process with a
>> single thread -- a copy of the one that called fork().
>> 
> Sorry to ask...what do you mean by `the "UI thread" version of fork()'?
> I'm a little confused here.

Alright, if you're only "a little confused", then we haven't done our jobs. 
We'll try for "very confused", OK? Let me know when we're there. ;-)

First, the reference to "UI threads" may have seemed to come out of the 
blue if you're new to this newsgroup and threads; so let's get that out of 
the way. "UI" was a committee that for a time controlled the direction and 
architecture of the System V UNIX specification. (UNIX International.) The 
thread interfaces and behavior they defined (which was essentially what Sun 
had devised for Solaris, modified somewhat along POSIX lines in places) are 
commonly known as "UI threads". (Or sometimes "Solaris threads" since they 
originated on Solaris and aren't widely available otherwise.)

The UI thread definition of fork() is that all threads exist, and continue 
execution, in the child process. Threads that are blocked, at the time of 
the fork(), in a function capable of returning EINTR *may* do so (but need 
not). The problem with this is that fork() in a process where threads work 
with external resources may corrupt those resources (e.g., writing 
duplicate records to a file) because neither thread may know that the 
fork() has occurred. UI threads also has fork1(), which creates a child 
containing only a copy of the calling thread. This is equivalent to the 
POSIX fork() function, which provides a more controlled environment. (You 
can always use pthread_atfork() handlers to create daemon threads, or 
whatever else you want, in the child.)



=============================TOP===================================
 Q121: Behavior of [pthread_yield()] sched_yield() 

> > I have a question regarding POSIX threads on Linux and Solaris. The
> > program below compiles and links well on both systems, but instead of the
> > expected "100000, " it always prints out
> > "100000, 0", so the thread is not really ever started.
> 
> 
> 
> Well, both sets of output are legal and correct for the code you supplied.

Yes, this is correct.

> First, you see [p]thread_yeild does not say "give control to another thread"
> it says "if there is another thread that can be run, now might be a good time
> do do that".  The library is under no obligation to actually yeild.  (there
> is a good explaination of this elsewhere in this group, but it has to do with
> the fact that you are running under SCHED_OTHER semantics which are
> completely unspecified semantics, go figure.)

Just for clarity...

	pthread_yield is an artifact of the obsolete and crufty old
	DCE thread implementation (loose interpretation of the 1990
	draft 4 of the POSIX thread standard). It doesn't exist in
	POSIX threads.

	thr_yield is an artifact of the UI threads interface, which
	is, (effectively though not truly), Solaris proprietary.

	sched_yield is the equivalent POSIX function.

As Robert said, POSIX assigns no particular semantics to the SCHED_OTHER
scheduling policy. It's just a convenient name. In the lexicon that we
developed during the course of developing the realtime and thread POSIX
standards, it is "a portable way to be nonportable". When you use
SCHED_OTHER, which is the default scheduling policy, all bets are off.
POSIX says nothing about the scheduling behavior of the thread.
(Although it does require a conforming implementation to DOCUMENT what
the behavior will be.)

Because there's no definition of the behavior of SCHED_OTHER, it would
be rather hard to provide any guarantees about the operation of the
sched_yield function, wouldn't it?

If you want portable and guaranteed POSIX scheduling, you must use the
SCHED_FIFO or SCHED_RR scheduling policies (exclusively). And, of
course, you need to run on a system that supports them.

> Next, the number of threads in a (POSIX) program does not necessarily say
> anthing about the number of actual lightweight processes that will be used to
> execute the program.  In your example there is nothing that "forcably" causes
> the main thread to give up the processor (you are 100% CPU related) so your
> first thread runs through to completion.  An identically arranged ADA program
> (which wouldn't quite be possible 8-) would have equally unstable results.
> (I've seen students write essentially this exact program to "play with" tasks
> and threads in ADA and C, but the program is not valid in any predictable
> way.)

POSIX doesn't even say that there's any such thing as a "light weight
process". It refers only obliquely to the hypothetical concept of a
"kernel execution entity", which might be used as one possible
implementation mechanism for Process Contention Scope thread scheduling.

> Finally, POSIX only says that there will be "enough" LWPs at any moment to
> ensure that the program as a whole "continues to make progress".

That's not strictly true. All POSIX says is that a thread that blocks
must not indefinitely prevent other threads from making progress. It
says nothing about LWPs, nor places any requirements upon how many there
must be.

> When you do the SIGINT from the keyboard you are essentially causing the
> "current" thread to do a pthread_exit/abort. Now there is only one thread
> left, the "second" one, so to keep the program progressing that one get's the
> LWP from the main thread.  That is why you see the second start up when you
> do a "^C"...

SIGINT shouldn't "do" anything to a thread, on a POSIX thread system. IF
it is not handled by a sigaction or a sigwait somewhere in the process,
the default signal action will be to terminate the process (NOT the
thread).

It's not clear from the original posting exactly where the described
results were seen: Linux or Solaris? My guess is that this is Linux,
with the LinuxThreads package. Your threads are really cloned PROCESSES,
and I believe that LinuxThreads still does nothing to properly implement
the POSIX signal model among the threads that compose the "process".
That may mean that, under some circumstances, (and in contradiction to
the POSIX standard), a signal may affect only one thread in the process.
The LinuxThreads FaQ says that SIGSTOP/SIGCONT will affect only the
targeted thread, for example. Although it also says that threads "dying"
of a signal will replicate the signal to the other threads, that might
not apply to SIGINT, or there might be a timing window or an outright
hole where that's not happening in this case.

LinuxThreads is, after all, a freeware thread package that's from all
reports done an excellent job of attacking a fairly ambitious goal. A
few restrictions and nonconformancies are inevitable and apparently
acceptable to those who use it (although it's gotta be a portability
nightmare for those who use signals a lot, you're always best off
avoiding signals in threaded programs anyway -- a little extra
"incentive" isn't a bad thing). If you see this behavior on Solaris,
however, it's a serious BUG that you should report to Sun.

> The very same program with a single valid "operational yeild" (say reading a
> character from the input device right after the pthread_create()) will run at
> 100% CPU forever because it will never switch *OUT* of the second thread.

At least, that's true on Solaris, where user threads aren't timesliced.
To get multiple threads to operate concurrently, you need to either
manually create additional LWPs (thr_setconcurrency), or create the
threads using system contention scope (pthread_attr_setscope) so that
each has its own dedicated LWP. Solaris will timeslice the LWPs so that
multiple compute-bound threads/processes can share a single processor.
LinuxThreads directly maps each "POSIX thread" to a "kernel thread"
(cloned process), and should NOT suffer from the same problem. The
kernel will timeslice the "POSIX threads" just as it timeslices all
other processes in the system. On Digital UNIX, the 2-level scheduler
timeslices the user ("process contention scope") threads, so, if a
compute-bound SCHED_OTHER thread runs for its full quantum, another
thread will be given a chance to run.

> In essence there is no good "Hello World" program for (POSIX) threads (Which
> is essentially what you must have been trying to write 8-).  If the threads
> don't interact with the real world, or at least eachother, the overall
> program will not really run.  The spec is written to be very responsive to
> real-world demands.  That responsiveness in the spec has this example as a
> clear degenerate case.

That's not true. "Hello world" is easy. If the thread just printed
"Hello world" and exited, and main either joined with it, or called
pthread_exit to terminate without trashing the process, you'd see
exactly the output you ought to expect, on any conforming POSIX
implementation. The problem is that the program in question is trying to
execute two compute-bound threads concurrently in SCHED_OTHER policy:
and the behavior of that case is simply "out of scope" for the standard.
The translation of which is that there's no reasonable assumption of a
portable behavior.

/---------------------------[ Dave Butenhof ]--------------------------\


=============================TOP===================================
 Q126: Cancellation and condition variables  

Marcel Bastiaans wrote:

> Anyone:
>
> I appear to be missing something in my understanding of how condition
> variables work.  I am trying to write a multithreaded program which is
> portable to various platforms.  I am unable to cancel a thread if it is
> waiting on a condition variable which another thread is waiting on also.
> The problem can easily be reproduced on both Solaris 2.5 and HP-UX 10.10.  A
> simple program which demonstrates my problem is shown below.  This program
> sample uses the HP-UX pthreads library but the problem also appears when
> using Solaris threads on Solaris 2.5.

In any case... yes, you are missing something. The program, as written, will
hang on any conforming (or even reasonably correct) implementation of either
DCE threads or POSIX threads. (To put it another way, any implementation on
which it succeeds is completely broken.)

> Is there a problem in this program which I don't understand?  I cannot use
> cleanup handlers because not all platforms support them.  Any help would be
> greatly appreciated.

If you can use cancellation, you can use cleanup handlers. Both are part of
both DCE threads (what you're using on HP-UX 10.10) and POSIX threads (what you
probably are, and, at least, should be, using on Solaris 2.5.) If you've got
cancellation, and you don't have cleanup handlers, you've got an awesomely
broken implementation and you should immediately chuck it.

When you wait on a condition variable, and the thread may be cancelled, you
MUST use a cleanup handler. The thread will wake from the condition wait with
the associated mutex locked -- even if it was cancelled. If the thread doesn't
then unlock the mutex before terminating, that mutex cannot be used again by
the program... it will remain locked by the cancelled thread.

> #include 
> #include 
> #include 
>
> pthread_cond_t cond;
> pthread_mutex_t mutex;
>
> void * func(void *)
> {
>    // Allow this thread to be cancelled at any time
>    pthread_setcancel(CANCEL_ON);
>    pthread_setasynccancel(CANCEL_ON);

Serious, SERIOUS bug alert!! DELETE the preceding line before proceeding with
this or any other program. Never, ever, enable async cancelation except on
small sections of straight-line code that does not make any external calls.
Better yet, never use async cancel at all.

In any case, you absolutely CANNOT call any POSIX (or DCE) thread function with
async cancellation enabled except the ones that DISABLE async cancel. (For
bizarre and absolutely unjustifiable reasons [because they're wrong], POSIX
threads also allows you to call pthread_cancel -- but don't do it!)

>    // Wait forever on the condition var
>    pthread_mutex_lock(&mutex);
>    for(;;) {
>       pthread_cond_wait(&cond, &mutex);
>    }
>    pthread_mutex_unlock(&mutex);
>    return 0;
> }

I suspect your problem is in cancelling the second thread. As I said,
cancellation terminates the condition wait with the associated mutex locked.
You're just letting the thread terminate with the mutex still locked. That
means, cancelled or not, the second thread can never awaken from the condition
wait. (At a lower level, you could say that it HAS awakened from the condition
wait, but is now waiting on the mutex... and a mutex wait isn't cancellable.)

The answer is... if you use cancellation, you must also use cleanup handlers.
(Or other, non-portable equivalent mechanisms, such as exception handlers or
C++ object destructors... on platforms where they're implemented to
interoperate with cancellation. [Both Solaris and Digital UNIX, for example,
run C++ destructors on cancellation.])


=============================TOP===================================
 Q128: How do I measure thread timings?  
Andy Sunny wrote:

> I'm conducting some research to measure the following things about
> pthreads using a Multikron II Hardware Instrumentation Board from NIST
> 1) thread creation time (time to put thread on queue)
> 2) thread waiting time (time that thread waits on queue)
> 3) thread execution time (time that thread actually executes)
>
> Are there any decent papers that explain the pthreads run time system
> and scheduling policy in DETAIL? I have read Frank Mueller's (FSU) paper
> and am trying to obtain the standard from IEEE. What is the latest
> version of the standard and will it help me find the proper libraries
> and functions need to measure the above items?

The standard is unlikely to be of any help to you. It says nothing at all
about implementation. POSIX specifies SOURCE-LEVEL interfaces, and
describes the required portable semantics of those interfaces.
Implementation details are (deliberately, properly, and necessarily) left
entirely to the creator of each implementation. For example, there's no
mention of libraries -- an embedded system, for example, might include all
interfaces in an integrated kernel; and that's fine.

What you need is a document describing the internal implementation details
of the particular system you're using. If the vendor can't supply that,
you'll need to create it yourself -- either by reading source, if you can
get it, or by flailing around blindly in the dark and charting the walls
you hit.



=============================TOP===================================
 Q129: Contrasting Win32 and POSIX thread designs  
Arun Sharma wrote:

> On Mon, 24 Nov 1997 18:10:13 GMT, Christophe Beauregard wrote:
>
>         c>  while thread context is a Windows concept.
>
> How so ? pthreads don't have contexts ?

This looks like an interesting discussion, of which I've missed the
beginning. (Perhaps only the followup was cross-posted to
comp.programming.threads?) Anyway, some comments:Anything has "context".
A thread is an abstraction of the executable state traditionally
attributed to a process. The "process" retains the non-executable state,
including files and address space. Why would anyone contend that "thread
context is a Windows concept"? I can't imagine. Maybe it's buried in the
unquoted portion of the orignal message. And then again, some people
think Microsoft invented the world.

>         c> Generally, you'll find that pthreads gives you less control
>         c> over how a thread runs.  There are very good reasons for
>         c> this (one being portability, another being safety).
>
> In other words, it has to be the least common denominator in the
> fragmented UNIX world. No wonder people love NT and Win32 threads.

POSIX threads gives you far more real control over threads than Win32
(for example, far superior realtime scheduling control). What it doesn't
give you is suspend/resume and uncontrolled termination. Those aren't
"control over how a thread runs". They are extraordinarily poor
programming mechanisms that can almost never be used correctly. Yes, to
some people the key is "almost never", and one may argue that they should
be provided anyway for that 0.001% of applications that "need" it. (But
those of us who actually support threaded interfaces might also point out
that these extremely dangerous functions are for some reason particularly
tempting to beginners who don't know what they're doing -- resulting in
very high maintenance costs, which mostly involves helping them debug
problems in their code.)

This isn't an example of "fragmented UNIX" -- it's UNIX unity, with a
wide variety of different "UNIX camps" reaching a concensus on what's
necessary and useful.

While the Win32 interface comprises whatever the heck a few designers
felt like tossing in, POSIX was carefully designed and reviewed by a
large number of people, many of whom knew what they were doing. Omitting
these functions was a carefully considered, extensively discussed, and
quite deliberate decision. The Aspen committee that designed the thread
extensions to POSIX for the Single UNIX Specification, Version 2,
proposed suspend/resume -- they were later retracted by the original
proposer (with no objections). A POSIX draft standard currently under
development, 1003.1j, had proposed a mechanism for uncontrolled
termination, with the explicit recognition that it could be used (and
then only with extreme care) only in carefully constructed embedded
systems. It, too, was later withdrawn as the complications became more
obvious. (The notion that you can regain control of a process when you've
lost control of any one thread in the process is faulty, because all
threads depend completely on shared resources. If you've lost control of
a thread, you don't know the state of the process -- how can you expect
it to continue?)

>         c> Basically, using signals for dealing with threads is a Bad
>         c> Thing and people who try generally get screwed.
>
> It doesn't have to be so. That's an implementation problem.

Yes, it does have to be so, because signals are a bad idea to begin with.
Although there were enormous complications even before threads, the
concept becomes all but unsupportable with the addition of full
asynchronous execution contexts to the traditional process.

The "synchronous" signals, including SIGSEGV, should be language
exceptions. The other "asynchronous" signals should be handled
synchronously in independent contexts (threads). If you think about it,
that's what signals were attempting to do; the condition exists as a
separate execution context (the signal handler). Unfortunately, a signal
preempts the hardware context of the main execution context,
asynchronously. That's a really, really bad idea. Although people have
always casually done things like calling printf in signal handlers, too
few people realize that's always been incorrect and dangerous -- only a
small list of UNIX functions are "async-signal safe". The addition of
threads, however, allowing the process to have multiple contexts at any
time, increases the chances that some thread will be doing something that
will conflict with improper use of non async-signal safe functions at
signal level.

> Portability doesn't necessarily have to cripple the API.

And, in fact, it doesn't. It results in a well-designed and robust
interface that can be efficiently implemented everywhere. I'm not arguing
that the POSIX interface is perfect. There is room for additions, and the
Single UNIX Specification, Version 2, makes a good start. Other areas to
consider for future standardization would include debugging and analysis
interfaces. There are POSIX standards in progress to improve support for
"hard realtime" environments (for example, putting timeouts on all
blocking functions to control latency and help diagnose failures).


=============================TOP===================================
 Q143: Pthreads and Linux  

Wolfram Gloger wrote:


> You should always put `-lpthread' _after_ your source files:
>
> % gcc the_file.cpp -lpthread
>
> Antoni Gonzalez Ciria  writes:
>
> > When I compile this code with gcc ( gcc -lpthread the_file.cpp) the 
> > program executes fine, but doing so with g++( g++ -lpthread the_file.cpp)
> > the progran crashes, giving a Segmentation fault error.
> >
> > #include 
> >
> > void main(){
> >     FILE * the_file;
> >     char sBuffer[32];
> >
> >     the_file=fopen("/tmp/dummy","rb");
> >     fread( sBuffer, 12, 1, the_file);
> >     fclose( the_file);
> >
> > }
>
> Using `g++' as the compiler driver always includes the `libg++'
> library implicitly in the link.  libg++ has a lot of problems, and is
> no longer maintained (in particular, it has a global constructor
> interfering with glibc2, if I remember correctly).  If you really need
> it, you must get a specially adapted version for Linux/egcs/gcc-2.8.
>
> If you don't need libg++, please use `c++' or `gcc' as your compiler
> driver, and use the libstc++ library shipped with egcs or seperately
> with gcc-2.8 (`c++' will link libstdc++ in implicitly).
>
> When I just tested your program with egcs-1.0.1 and glibc-2.0.6, it
> crashed at first (it fails to check the fopen() result for being
> NULL), but after creating a /tmp/dummy file it ran perfectly, even
> after compiling it with `c++ the_file.cpp -lpthread'.
>
> Regards,
> Wolfram.

  The -pthread option takes care of everything:
     adds  -D_REENTRANT  during the cpp pass, and
     adds  -lpthread during the link-edit.
  This option has been around for a while. I'm not sure
   it's working for all ports. At least for the x86 AND glibc.
   You may want to take a look at the spec file (gcc -v)

jms.

=============================TOP===================================
 Q153:  C++ Exceptions in Multi-threaded Solaris Process  

Jeff Gomsi  writes:

> We are running a large multi-threaded C++ (C++ 4.2 patch 
> 104631-03) application under Solaris (SunOS 5.5.1 
> Generic_103640-14) on a 14 processor Ultra-Enterprise and 
> observe the following problem.
> 
> The application runs fine single-threaded, but when run
> multi-threaded, throwing a C++ exception can (evidently) 
> cause memory corruption which leads to a SIGSEGV core
> dump. A diagnostic version of the new operator seems to
> reveal that C++ is unwinding things improperly and possibly
> calling destructors which should not be called.
> 
> Does anyone have any ideas on this?

The last time I looked at the patch list for the C++ 4.2, I noticed a
mention of a bug stating that exceptions were not thread safe.  There was
no further description of this bug that I could find.  However, it
supposedly is addressed by one of the later patches. Try upgrading your
patch to -04 or -05....

- Chris

Make sure you have the libC patch 101242-13.

=============================TOP===================================
 Q155: "lock-free synchronization"  

> I recently came across a reference to "lock-free synchronization" (in
> Taligent's Guide to Designing Program's.)  This document referred to
> research that was looking at using primitive atomic operations to build more
> complex structures in ways that did not require locking.
>
> I'm interested in exploring this topic further and would be greatful if
> anyone could supply references.
>
> Regards,
> Daniel Parker
>
>

Check out the following references --

  M. Herlihy, "Wait free Synchronization," ACM Transactions on Programming
Languages and Systems, Vol 13, No 1, 1991, pp. 124-149.

  M. Herlihy, "A Methodology for Implementing Highly Concurrent Data
Objects," same journal as above, Vol 15, No. 5, 1993, pp. 745 --770.

They should provide a starting point.

=============================TOP===================================
 Q156: Changing single bytes without a mutex  

Tim Beckmann wrote:

> David Holmes wrote:
> >
> > I thought about this after posting. An architecture such as Bil describes
> > which requires non-atomic read/mask/write sequences to update variables of
> > a smaller size than the natural word size, would be a multi-threading
> > nightmare. As you note above two adjacent byte values would need a common
> > mutex to protect access to them and this applies even if they were each
> > used by only a single thread! On such a system I'd only want to program
> > with a thread-aware language/compiler/run-time.
> >
> > David
>
> David,
>
> My thoughts exactly!
>
> Does anyone know of a mainstream architecture that does this sort of
> thing?

Oh, absolutely. SPARC, MIPS, and Alpha, for starters. I'll bet most other RISC
systems do it, too, because it substantially simplifies the memory subsystem
logic. And, after all, the whole point of RISC is that simplicity means speed.

If you stick to int or long, you'll probably be safe. If you use anything
smaller, be sure they're not allocated next to each other unless they're under
the same lock.

I wrote a long post on most of the issues brought up in this thread, which
appears somewhere down the list due to the whims of news feeds, but I got
interrupted and forgot to address this issue.

If you've got

     pthread_mutex_t mutexA = PTHREAD_MUTEX_INITIALIZER;
     pthread_mutex_t mutexB = PTHREAD_MUTEX_INITIALIZER;

     char dataA;
     char dataB;

And one thread locks mutexA and writes dataA while another locks mutexB and
writes dataB, you risk word tearing, and incorrect results. That's a "platform
issue", that, as someone else commented, POSIX doesn't (and can't) address.

What do you do? I always advise that you keep a mutex and the data it protects
closely associated. As well as making the code easier to understand, it also
addresses problems like this. If the declarations were:

     typedef struct dataTag {
         pthread_mutex_t mutex;
         char data;
     } data_t;

     data_t dataA = {PTHREAD_MUTEX_INITIALIZER, 0};
     data_t dataB = {PTHREAD_MUTEX_INITIALIZER, 1};

You can now pretty much count on having the two data elements allocated in
separate "memory access chunks". Not an absolute guarantee, since a
pthread_mutex_t might be a char as well, and some C compilers might not align
structures on natural memory boundaries. But most compilers on machines that
care WILL align/pad structures to fit the natural data size, unless you
override it with a switch or pragma (which is generally a bad idea even when
it's possible). And, additionally, a pthread_mutex_t is unlikely to be less
than an int, and is likely at least a couple of longs. (On Digital UNIX, for
example, a pthread_mutex_t is 48 bytes, and on Solaris it's 24 bytes.)

There are, of course, no absolute guarantees. If you want to be safe and
portable, you might do well to have a config header that typedefs
"smallest_safe_data_unit_t" to whatever's appropriate for the platform. Then
it's just a quick trip to the hardware reference manual when you start a port.
On a CISC, you can probably use "char". On most RISC systems, you should use
"int" or "long".

Yes, this is one more complication to the process of threading old code. But
then, it's nothing compared to figuring out which data is shared and which is
private, and then getting the locking protocols right.

/---------------------------[ Dave Butenhof ]--------------------------\
| Digital Equipment Corporation                   butenhof@zko.dec.com |
| 110 Spit Brook Rd ZKO2-3/Q18       http://members.aol.com/drbutenhof |
| Nashua NH 03062-2698  http://www.awl.com/cseng/titles/0-201-63392-2/ |
\-----------------[ Better Living Through Concurrency ]----------------/

> If I'm not mistaken, isn't that spelled:
>
>     #include 
>
>     typedef sig_atomic_t smallest_safe_data_unit_t;

You are not mistaken, and thank you very much for pointing that out. While I'd
been aware at some point of the existence of that type, it was far from the top
of my mind.

If you have data that you intend to share without explicit synchronization, you
should be safe in using sig_atomic_t. Additionally, using sig_atomic_t will
protect you against word tearing in adjacent data protected by separate mutexes.

There are additional performance considerations, such as "false sharing" effects
in cache systems, that might dictate larger separations between two shared pieces
of data: but those won't affect program CORRECTNESS, and are therefore more a
matter of tuning for optimal performance on some particular platform.


=============================TOP===================================
 Q158: VOLATILE instead of mutexes?  


> What about exception handlers ? I've always thought that when you had
> code like:
>
>         int i;
>
>         TRY
>         {
>                 
>                 . . .
>                 proc();
>         }
>         CATCH_ALL
>         {
>                 if (i > 0)
>                 {
>                         . . .
>                 }
>                 . . .
>         }
>
> that you needed to declare "i" to be volatile least the code in the
> catch block assume that "i" was stored in some register the contents
> of which were overwritten by the call to "proc" (and not restored by
> whatever mechanism was used to throw the exception).

Since neither ANSI C nor POSIX has any concept remotely resembling "exceptions", this
is all rather moot in the context of our general discussion, isn't it? I mean, it's
got nothing to do with sharing data between threads -- and that's what I thought we
were talking about. But sure, OK, let's digress.

Since there's no standard covering the behavior of anything that uses exceptions, (at
least, not if you use them from C, or even if you use the DCE exception syntax you've
chosen from C++), there's no portable behavior. Your fate is in the hands of the
whimsical (and hypothetically malicious ;-) ) implementation. This situation might
lead a cautious programmer to be unusually careful when treading in these waters, and
to wear boots with thick soles. (One might also say that it could depend on exactly
HOW you "foodle with i", but I'll choose to disregard an entire spectrum of mostly
amusing digressions down that fork.)

Should you use volatile in this case? Sure, why not? It might not be necessary on
many platforms. It might destroy your performance on any platform. And, where it is
necessary, it might not do what you want. But yeah, what the heck -- use it anyway.
It's more likely (by some small margin) to save you than kill you.

Or, even better... don't code cases like this!

/---------------------------[ Dave Butenhof ]--------------------------\

=============================TOP===================================
 Q167: Using cancellation is *very* difficult to do right...  

Bil Lewis wrote:

> Dave Butenhof wrote:
> > >   Using cancellation is *very* difficult to do right, and you
> > > probably don't want to use it if there is any other way you can
> > > accomplish your goal.  (Such as looking at a "finish" flag as you
> > > do below.)
> >
> > I don't agree that cancellation is "very" difficult, but it does
> > require understanding of the application, and some programming
> > discipline. You have to watch out for cancellation points, and be
> > sure that you've got a cleanup handler to restore shared data
> > invariants and release resources that would otherwise be stranded if
> > the thread "died" at that point. It's no worse than being sure you
> > free temporary heap storage, or unlock your mutexes, before
> > returning from a routine... but that's not to say that it's trivial
> > or automatic. (And I'm sure we've never gotten any of those things
> > wrong... ;-) )
>
>   Dave has written 10x as much UNIX code as I have, so our definitions
> of "very difficult" are distinct.  (I've probably been writing MP code
> longer tho...  I built my first parallel processor using two PDP/8s
> back in '72.  Now THERE was a RISC machine!  DEC could have owned the
> world if they'd tried.  I remember when...)

Yeah, PDP-8 was a pretty good RISC, for the time. Of course it needed
virtual memory, and 12 bits would now be considered a rather "quirky"
word size. But, yeah, those could have been fixed.

Oh yeah... and we DID own the world. We just let it slip out of our
hands because we just laughed when little upstarts said they owned it.
(Unfortunately, people listened, and believed them, and eventually it
came to be true.) ;-) ;-)

>   It's that bit "to restore shared data invariants". Sybase, Informix,
> Oracle, etc. spend years trying to get this right.  And they don't
> always succeed.

It's hard to do hard things. Yeah, when you've got an extremely
complicated and extremely large application, bookkeeping gets more
difficult. This applies to everything, not just handling cancellation.
Just as running a multinational corporation is harder than running a
one-person home office. The point is: the fact that the big job is hard
doesn't mean the small job is hard. Or, you get out what you put in. Or
"thermodynamics works". Or whatever.

>   And don't forget to wait for the dead threads.  You can't do
> anything with the shared data until those have all been joined,
> because you can't be sure when they actually die.

That's untrue, as long as you use proper synchronization (or maybe "as
long as you use synchronization properly"). That's exactly why the mutex
associated with a condition wait is re-locked even when the wait is
cancelled. Cleanup code needs (in general) to restore invariants before
the mutex can be safely unlocked. (Note that while the data must be
"consistent", at the time of the wait, because waiting has released the
mutex, it's quite common to modify shared data in some way associated
with the wait, for example, adding an item to a queue; and often that
work must be undone if the wait terminates.)

You only need to wait for the cancelled thread if you care about it's
return value (not very interesting in this case, since it's always
PTHREAD_CANCELED, no matter how many times you look), or if you really
want to know that it's DONE cleaning up (not merely that the shared data
is "consistent", but that it conforms to some specific consistency -- an
attempt that I would find suspicious at best, at least if there might be
more than the two threads wandering about), or if you haven't detached
the thread and want to be sure it's "gone".

>   Conclusion: Dave is right (of course).  The definition of "very" is
>   up for grabs.

The definition of the word "very" is always up for grabs. As Samuel
Clemens once wrote, when you're inclined to use the word "very", write
"damn" instead; your editor will remove it, and the result will be
correct.

Sure, correct handling of cancellation doesn't happen automatically.
Neither does correct use of threads, much less correct use of the arcane
C language (and if C is "arcane", what about English!?) Somehow, we
survive all this.

/---------------------------[ Dave Butenhof ]--------------------------\
| Digital Equipment Corporation                   butenhof@zko.dec.com |
| 110 Spit Brook Rd ZKO2-3/Q18       http://members.aol.com/drbutenhof |
| Nashua NH 03062-2698  http://www.awl.com/cseng/titles/0-201-63392-2/ |
\-----------------[ Better Living Through Concurrency ]----------------/


=============================TOP===================================
 Q210: Where is the threads standard of POSIX ????  

try http://www.unix-systems.org/single_unix_specification_v2/xsh/threads.html
 

=============================TOP===================================
 Q211: Is Solaris' unbound thread model braindamaged?  

"Doug Royer [N6AAW]" wrote:

> Did you have a specifc braindamaged bug to report?
>
> In article <364AF6D0.EAB3A347@ms.com>, Boris Goldberg  writes:
> >
> > I briefly browsed Solaris 7 docs at docs.sun.com and, regrettably,
> > it doesn't appear that they changed their braindamaged threading model.

Actually, I think Doug phrased that very well. In particular, he didn't use the
word "bug". He merely said "braindamaged". One might easily infer, (as I have),
that he's making the assumption that the "braindamaged" behavior is intentional,
and simply expressing regret that the intent hasn't changed.

Here's a few of the common problems with Solaris 2-level threading. I believe
one of them could accurately be described as a "bug" in Solaris (and that's not
confirmed). The others are merely poor design decisions. Or, in common terms,
"brain damage".

  1. thr_concurrency() is a gross hack to avoid implementing most of the 2-level
     scheduler. It means the scheduler puts responsibility for maintaining
     concurrency on the programmer. Nice for the Solaris thread subsystem
     maintainers -- not so nice for users. (Yes, UNIX has a long and
     distinguished history of avoiding kernel/system problems by complicating
     the life of all programmers. Not all of those decisions are even wrong.
     Still, I think this one is unnecessary and unjustifiable.)
  2. Rumor has suggested that Solaris creates one LWP by default even on SMP
     systems -- if that rumor is true, this condition might shade over the line
     into "true bug". But then, having an SMP isn't necessarily the same as
     being able to use it, so maybe that's deliberate, too.
  3. Blocking an LWP reduces the process concurrency. Yeah, sure the library
     will create a new one when the last LWP blocks, but that's not good. First,
     it means the process has been operating on fewer cylinders than it might
     think for some period of time. And, in many cases even worse, after the
     LWPs unblock, it will be operating on more cylinders than it can sustain
     until the LWPs go idle and time out. Running with more LWPs than processors
     is rarely a good idea unless most of them will always be blocked in the
     kernel. (I've heard unsubstantiated rumors that 2.6 did some work to
     improve on this, and 7 may do more; but I'm not inclined to let anyone "off
     the hook" without details.)
  4. While timeslicing is not required by POSIX, it is the scheduling behavior
     all UNIX programmers (and most who are used to other systems, as well)
     EXPECT. The lack of timeslicing in Solaris 2-level scheduling is a constant
     source of complication and surprise to programmers. Again, this isn't a
     bug, because it's clearly intentional; it's still a bad idea, and goes
     against the best interests of application programmers.

/---------------------------[ Dave Butenhof ]--------------------------\
| Compaq Computer Corporation                     butenhof@zko.dec.com |
| 110 Spit Brook Rd ZKO2-3/Q18       http://members.aol.com/drbutenhof |
| Nashua NH 03062-2698  http://www.awl.com/cseng/titles/0-201-63392-2/ |
\-----------------[ Better Living Through Concurrency ]----------------/

 


=============================TOP===================================
 Q212:  Releasing a mutex locked (owned) by another thread.  
Zoom wrote:

> Hello, I have inherited the maintenance of a multi-threaded application.
> The application uses pthreads and runs on multiple platforms including
> solaris. On solaris it seems to be somewhat squirrely (the technical
> term of course :-) and I get random core dumps or thread panics.
> Absolutely not consistantly reproduceable. Sometimes it will go for
> hours or days cranking away and sometimes it will thread panic shortly
> after it starts up. In researching the the book "Multi-threaded
> Programming with Pthreads" by Bil Lewis et. al. I found on page 50 the
> statement to the effect that under posix it is illegal for one thread to
> release a mutex locked (owned) by another thread. Well, this application
> does that. In fact it does it quite extensively.
>
> Is there anyone willing to commit to the idea that this may be the
> source of the applications problems.

The answer is an absolutely, definite, unqualified "maybe". It depends
entirely on what the application is doing with those mutexes.

First, I want to be completely clear about this. Make no mistake, locking a
mutex from one thread and unlocking it from another thread is absoutely
illegal and incorrect. The application is seriously broken, and must be
fixed.

However, reality is a little more complicated than that. POSIX explicitly
requires that application programmers write correct applications. More
specifically, should someone write an incorrect application, it explicitly
and deliberately does NOT require that a correct implementation of the
POSIX standard either DETECT that error, or FAIL due to that error. The
results of programmer errors are "undefined". (This is the basis of the
POSIX standard wording on error returns -- there are "if occurs" errors,
which represent conditions that the programmer cannot reasonably
anticipate, such as insufficient resources; and there are "if detected"
errors, which are programmer errors that are not the responsibility of the
implementation. A friendly/robust implementation may choose to detect and
report some or all of the "if detected" errors -- but even when it fails to
detect the error, it's still the application's fault.)

The principal difference between a binary semaphore and a mutex is that a
mutex carries with it the concept of "ownership". It is that characteristic
that makes it illegal to unlock the mutex from another thread. The locking
thread OWNS the mutex, exclusively, until it unlocks the mutex. IF an
implementation can (and chooses to) detect and report violations of the
ownership protocol, the erroneous attempt at unlock will result in an EPERM
return. However, this is a programmer error. It is often unreasonably
expensive to keep track of which thread owns a mutex: an instruction (or
kernel call) to determine the identity of the locking thread may take far
longer than the basic lock operation. And of course it would be equally
expensive to check for ownership during unlock.

Many implementations of POSIX threads, therefore, do not record, or check,
mutex ownership. However, because it's a mutex, it IS owned, even if the
ownership isn't recorded. The next patch to your operating system might add
checking, or it might be possible to run threaded applications in a
heavyweight debug environment where mutex ownership is recorded and
checked... and the erroneous code will break the application. It'll be the
application's (well, the application developer's) fault.

Anyway, IF the implementation you're using really doesn't record or check
ownership of mutexes. And IF that illegal unlock is done as part of a
carefully managed "handoff" protocol so that there's no chance that the
owner actually needs the mutex for anything. (And, of course, if this
bizarre and illegal protocol is actually "correct" and consistent.) THEN,
your application should work despite the inherent illegality.

You could switch to a binary semaphore, and do the same thing without the
illegality. The application still won't WORK if you're releasing a lock
that's actually in use.

/---------------------------[ Dave Butenhof ]--------------------------\
| Compaq Computer Corporation                     butenhof@zko.dec.com |
| 110 Spit Brook Rd ZKO2-3/Q18       http://members.aol.com/drbutenhof |
| Nashua NH 03062-2698  http://www.awl.com/cseng/titles/0-201-63392-2/ |
\-----------------[ Better Living Through Concurrency ]----------------/


=============================TOP===================================
 Q293: Details on MT_hot malloc()?  

There are a number of malloc() implemenations which scale better
than the simple, globally locked version used in Solaris 2.5 and
earlier. A good reference is:

http://www.ddj.com/articles/2001/0107/0107toc.htm

Some comments by the author:

If you quote them in the FAQ, make sure to make a note that these
opinions are my personal ones, not my employer's.

As I tried to describe in my DDJ article, there is no best malloc
for all cases.

To get the best version, I advise the application developers
to try different versions of mt-hot malloc with their specific
app and typical usage patterns and then select the version working
best for their case.

There are many mt-hot malloc implementations available now. Here
are my comments about some of them.

* My mt-hot malloc as described in the DDJ article and the patent.

It was developed first chronologically (as far as I know). It works
well when the malloc-hungry threads mostly use their own memory.
It also uses a few other assumptions described in my DDJ paper.

The main malloc algorithm in my mt-hot malloc is the same binary
search tree algorithm used in the default Solaris libc malloc(3C).

* mtmalloc(3t) introduced in Solaris 7.

I can't comment on this version, other than to say that it's
totally different from my mt-hot malloc implementation.

* Hoard malloc

It's famous, but my test (described in the DDJ article) did not
scale with Hoard malloc at all. It appeas that their realloc()
implementation is broken; at least it was in the version available
at the time of my testing (spring 2001). I've heard reports from
some Performance Computing people (who use Fortran 90 and no
realloc()) that Hoard malloc has helped their scalability very well.

Also, IMHO the Hoard malloc is too complicated, at least for the
simple cases using the assumptions described in my DDJ article.

* ptmalloc (a part of GNU libc)

I have not tested ptmalloc, so I can't comment on it.

* Smart Heap/SMP from MicroQuill

My tests of Smart Heap/SMP were not successful.

-Greg Nakhimovsky

=============================TOP===================================
 Q354: Using POSIX threads on mac X and solaris? 


>Does any one know of any advantages or disavtanges of using posix thread
>(pthread) on mac X and solaris compared to native implementations.
>
>Do pthread make call to native implementation in both these cases and is the
>maping between pthread and kernel object 1:1 .
>
>Thanks
>Sujeet
  
I don't know anything about the thread implementation on the mac. On Solaris,
pthreads are roughly equivalent to the so-called solaris threads
implementation. I believe that both APIs sit on top of lower-level calls.
The main advantage of using POSIX threads is portability. The other is
simplicity.

% Do pthread make call to native implementation in both these cases and is the
% maping between pthread and kernel object 1:1 .

The mapping between pthreads and the kernel scheduling entity in Solaris
depends on what you ask for. Note that you must be careful if you try to
use the m:n model, because the Solaris two-level thread scheduler is crap.
(this is not related to the API -- it's crap for both pthreads and UI threads).
--
 

On Mac OS X, POSIX threads is the lowest-level threading interface anyone
should be calling, at least outside the kernel. The POSIX interface uses Mach
threads, and there is an API to create Mach threads -- but it's not very
convenient. (You need to create the thread, load the registers with intimate
knowledge of the calling standard, including creating a stack and setting it to
"bootstrap" the thread.) Also, the Mach API has limited (and inefficient)
synchronization mechanisms -- IPC.

On Solaris, while libpthread depends on libthread, UI threads isn't really so
much a "native implementation"; they're more or less parallel, and happen to
share a common infrastructure, which happens (for mostly historical reasons) to
reside in libthread. You could consider the LWP layer to be "native threads",
but, somewhat like Mach threads, they're really not intended for general use.

The POSIX thread API is far more general, efficient, and portable than Mach,
UI, or LWP interfaces. Unfortunately, the POSIX thread implementation on Mac OS
X is incomplete, (it can't even build half of my book's example programs), and
I wouldn't want to count on it for much. (Though I have no evidence that what's
there doesn't work.) Still, you wouldn't be any better off working directly
with Mach threads.

Solaris, by the way, supports both "N to M" and "1 to 1" thread mappings.
Solaris 8 has a special library that's always 1 to 1. The normal libpthread
provides both N to M (Process Contention Scope, or PCS) and 1 to 1 (System
Contention Scope, or SCS); though the default is PCS and you can't change the
scope of the initial thread. Mac OS X supports only 1 to 1 scheduling.

/------------------[ David.Butenhof@compaq.com ]------------------\
| Compaq Computer Corporation              POSIX Thread Architect |
|     My book: http://www.awl.com/cseng/titles/0-201-63392-2/     |
\-----[ http://home.earthlink.net/~anneart/family/dave.html ]-----/

=============================TOP===================================