Sizing thread pools

from by : java concurrency in practice

8.2 Sizing thread pools
The ideal size for a thread pool depends on the types of tasks that will be submitted
and the characteristics of the deployment system. Thread pool sizes should
rarely be hard-coded; instead pool sizes should be provided by a configuration
mechanism or computed dynamically by consulting Runtime.availableProcessors.
Sizing thread pools is not an exact science, but fortunately you need only
avoid the extremes of “too big” and “too small”. If a thread pool is too big,
then threads compete for scarce CPU and memory resources, resulting in higher
memory usage and possible resource exhaustion. If it is too small, throughput
suffers as processors go unused despite available work.
To size a thread pool properly, you need to understand your computing environment,
your resource budget, and the nature of your tasks. How many processors
does the deployment system have? How much memory? Do tasks perform
mostly computation, I/O, or some combination? Do they require a scarce resource,
such as a JDBC connection? If you have different categories of tasks with
very different behaviors, consider using multiple thread pools so each can be
tuned according to its workload.
For compute-intensive tasks, an Ncpu-processor system usually achieves optimum
utilization with a thread pool of Ncpu +1 threads. (Even compute-intensive
threads occasionally take a page fault or pause for some other reason, so an “extra”
runnable thread prevents CPU cycles from going unused when this happens.)
For tasks that also include I/O or other blocking operations, you want a larger
pool, since not all of the threads will be schedulable at all times. In order to size
the pool properly, you must estimate the ratio of waiting time to compute time
for your tasks; this estimate need not be precise and can be obtained through profiling
or instrumentation. Alternatively, the size of the thread pool can be tuned
by running the application using several different pool sizes under a benchmark
load and observing the level of CPU utilization.

Given these definitions:
Ncpu = number of CPUs
Ucpu = target CPU utilization, 0 ≤ Ucpu ≤ 1
W/C = ratio of wait time to compute time
The optimal pool size for keeping the processors at the desired utilization is:
Nthreads = Ncpu ∗ Ucpu ∗ (1+ W/C)

You can determine the number of CPUs using Runtime:
int N_CPUS = Runtime.getRuntime().availableProcessors();


Of course, CPU cycles are not the only resource you might want to manage
using thread pools. Other resources that can contribute to sizing constraints are
memory, file handles, socket handles, and database connections. Calculating pool
size constraints for these types of resources is easier: just add up how much of
that resource each task requires and divide that into the total quantity available.
The result will be an upper bound on the pool size.
When tasks require a pooled resource such as database connections, thread
pool size and resource pool size affect each other. If each task requires a connection,
the effective size of the thread pool is limited by the connection pool size.
Similarly, when the only consumers of connections are pool tasks, the effective
size of the connection pool is limited by the thread pool size.