Resin 10,000

Resin v. Apache

Caveats

As always, no artificial benchmark can replace measuring your own application with your own configuration and load. In most cases, other considerations like the application performance and database performance will dominate the performance (although proxy caching can make slow applications run as fast as static pages.) These numbers in particular are a trivial tests with a simple load. They do measure the maximum performance of the web server, so they are valuable information, but they are very different from a benchmark of a complete application.

Warnings aside, these results do indicate that many sites should seriously consider using Resin as their web-tier load-balancing server. (After benchmarking your own application, of course.)

Heap size

Heap size does not determine the amount of memory your process uses

If you monitor your java process with an OS tool like top or taskmanager, you may see the amount of memory you use exceed the amount you have specified for -Xmx. -Xmx limits the java heap size, java will allocate memory for other things, including a stack for each thread. It is not unusual for the total memory consumption of the VM to exceed the value of -Xmx.

Garbage collection

Stack size

Monitoring the JVM

How many concurrent users can a Resin server handle?

How does Resin use JNI?

General

What are the best things to tune for better performance?

The main configuration item is the dependency-check-interval especially for Windows. For deployment, you should set it something high like 60s or larger.

You can also change the cache-mapping values, especially for stuff like *.gif files that don't change. Higher values mean that the browsers won't need to go back to the server.

Other than that, most of the default configuration values are pretty good, so you normally won't need to touch them.

The most important performance tweak you can make is to set Expires or better Last-Modified and/or ETag values on your servlet/JSP output. If the servlet/JSP output only changes every 15 minutes, as for a news page, then caching it can be a big performance win.

Of course, for stuff like shopping carts and stuff that's personalized, that won't help. But for many sites, the most heavily hit pages can be cached.

Is Apache faster than Resin Standalone?

For small files, Resin is about 10-20% faster. For large files (1M), they're essentially identical. (It's possible that the very latest Apache has improved performance.)

For JSP and Servlets, Resin standalone is certainly faster than Resin/Apache. Because of the extra overhead of the Resin/Apache connection, the Resin/Apache configuration is necessarily slower than Resin standalone.

It's only static files where Apache could be faster. Well, there's an exception for SSL. It's conceivable that Apache/Resin with SSL would be faster that Resin with SSL.

What is the performance loss with a Servlet or JSP comparted to a static file?

With Resin standalone, JSP files are essentially as fast as static files (as long as you don't actually do any processing. :-)

If Resin is behind another web server, like IIS or Apache, there is a performance decrease with JSP and Servlet files, which comes from the overhead needed for the communication between the other web server and Resin.

Caching

What gets cached when a servlet does a forward?

The only thing that matters is the HTTP headers. So if you telnet to the server, you should be able to see whether the headers are properly set or not.

In the case of a forward, you should be able to just set the headers without needing to modify the JSP itself.

One thing to be aware of: the caching is based on the original URL. So if your forwarding servlet varies it's output based on some request headers (like User-Agent), it needs to set the Vary header.

<cache-mapping> is a related but somewhat separate issue, and I think we haven't explained it properly.

<cache-mapping> only works on cacheable responses which have not set the Expires header. If you're missing the Expires header, <cache-mapping> will set it for you.

Cacheable means:

1. either ETag or Last-Modified must be set in the response (ETag is better). The servlet will normally set that value.
2. no cache-control is set in the response headers
3. no Vary tag is set (Resin doesn't completely implement Vary.)

So your servlet still needs to do some work. <cache-mapping> isn't all that you need. The reason that <cache-mapping> works with normal files is that Resin's FileServlet sets the Last-Modified and ETag headers, but does not set the Expires header.

What if while the cache is being filled, another request comes?

Resin 'fills a cache' the first time a request comes in. If another request comes in and Resin has not finished filling the cache, the second request will be treated as uncachable. This means that until the cache is filled, requests will miss the cache and get serviced directly.

This is also what happens when the cache expires. The first request to come in after the expiry time invalidates it, and while it is being filled the other requests pass through to the resource being cached.

This behaviour may be changed in Resin 3.0, updates are available here.

Resin Threads

Keepalives

Timeouts

Requests and keepalive connections can only be idle for a limited time before Resin closes them. Each connection has a read timeout, request-timeout. If the client doesn't send a request within the timeout, Resin will close the TCP socket. The timeout prevents idle clients from hogging Resin resources.

<resin xmlns="http://caucho.com/ns/resin">
  <cluster id="app-tier">

    <server-default>
      <thread-max>250</thread-max>

      <socket-timeout>30s</socket-timeout>

      <http port="8080"/>
    </server-default>

    ...
  </cluster>
</resin>

<resin xmlns="http://caucho.com/ns/resin">
  <cluster id="app-tier">

    <server-default>
      <thread-max>250</thread-max>

      <load-balance-idle-time>20s</load-balance-idle-time>

      <socket-timeout>30s</socket-timeout>

      <http port="8080"/>
    </server-default>

     <server id="app-a" address="192.168.2.1" port="6802"/>

     ...
   <cluster>
 <resin>

In general, the socket-timeout and keepalives are less important for Resin standalone configurations than Apache/IIS/srun configurations. Very heavy traffic sites may want to reduce the timeout for Resin standalone.

Since socket-timeout will close srun connections, its setting needs to take into consideration the <load-balance-idle-time setting for mod_caucho or isapi_srun. load-balance-idle-time is the time the plugin will keep a connection open. socket-timeout must always be larger than load-balance-idle-time, otherwise the plugin will try to reuse a closed socket.

Plugin keepalives (mod_caucho/isapi_srun)

The web server plugin, mod_caucho, needs configuration for its keepalive handling because requests are handled differently in the web server. Until the web server sends a request to Resin, it can't tell if Resin has closed the other end of the socket. If the JVM has restarted or if closed the socket because of socket-timeout, mod_caucho will not know about the closed socket. So mod_caucho needs to know how long to consider a connection reusable before closing it. load-balance-idle-time tells the plugin how long it should consider a socket usable.

Because the plugin isn't signalled when Resin closes the socket, the socket will remain half-closed until the next web server request. A netstat will show that as a bunch of sockets in the FIN_WAIT_2 state. With Apache, there doesn't appear to be a good way around this. If these become a problem, you can increase socket-timeout and load-balance-idle-time so the JVM won't close the keepalive connections as fast.

unix> netstat
...
localhost.32823      localhost.6802       32768      0 32768      0 CLOSE_WAIT
localhost.6802       localhost.32823      32768      0 32768      0 FIN_WAIT_2
localhost.32824      localhost.6802       32768      0 32768      0 CLOSE_WAIT
localhost.6802       localhost.32824      32768      0 32768      0 FIN_WAIT_2
...

TCP limits (TIME_WAIT)

A client and a server that open a large number of TCP connections can run into operating system/TCP limits. If mod_caucho isn't configured properly, it can use too many connections to Resin. When the limit is reached, mod_caucho will report "can't connect" errors until a timeout is reached. Load testing or benchmarking can run into the same limits, causing apparent connection failures even though the Resin process is running fine.

The TCP limit is the TIME_WAIT timeout. When the TCP socket closes, the side starting the close puts the socket into the TIME_WAIT state. A netstat will short the sockets in the TIME_WAIT state. The following shows an example of the TIME_WAIT sockets generated while benchmarking. Each client connection has a unique ephemeral port and the server always uses its public port:

unix> netstat
...
tcp   0   0 localhost:25033  localhost:8080  TIME_WAIT   
tcp   0   0 localhost:25032  localhost:8080  TIME_WAIT   
tcp   0   0 localhost:25031  localhost:8080  TIME_WAIT   
tcp   0   0 localhost:25030  localhost:8080  TIME_WAIT   
tcp   0   0 localhost:25029  localhost:8080  TIME_WAIT   
tcp   0   0 localhost:25028  localhost:8080  TIME_WAIT
...

The socket will remain in the TIME_WAIT state for a system-dependent time, generally 120 seconds, but usually configurable. Since there are less than 32k ephemeral socket available to the client, the client will eventually run out and start seeing connection failures. On some operating systems, including RedHat Linux, the default limit is only 4k sockets. The full 32k sockets with a 120 second timeout limits the number of connections to about 250 connections per second.

If mod_caucho or isapi_srun are misconfigured, they can use too many connections and run into the TIME_WAIT limits. Using keepalives effectively avoids this problem. Since keepalive connections are reused, they won't go into the TIME_WAIT state until they're finally closed. A site can maximize the keepalives by setting thread-keepalive large and setting live-time and request-timeout to large values. thread-keepalive limits the maximum number of keepalive connections. live-time and request-timeout will configure how long the connection will be reused.

<resin xmlns="http://caucho.com/ns/resin">
  <cluster id="app-tier">

    <server-default>
      <thread-max>250</thread-max>

      <keepalive-max>250</keepalive-max>
      <keepalive-timeout>120s</keepalive-timeout>

      <load-balance-idle-time>100s</load-balance-idle-time>
    </server-default>

    <server id="app-a" address="192.168.2.1"/>

    ...
  </cluster>
</resin>

socket-timeout must always be larger than load-balance-idle-time. In addition, keepalive-max should be larger than the maximum number of Apache processes.