Jive Community: Clearspace 2.0, Tomcat 5.5 service ...

Up to Discussions in Clearspace

This Question is Possibly Answered

1 "correct" answer available (4 pts) 2 "helpful" answers available (2 pts)

35 Replies Last post: May 28, 2008 6:56 AM by ryan.rutan@ni.com

casadyc

33 posts since
Apr 27, 2007

Apr 8, 2008 12:20 PM

Clearspace 2.0, Tomcat 5.5 service hangs on Windows

When I was upgraded from Clearspace 1.10 to 2.0 using the WAR file with a Tomcat 5.5 server on Windows 2003, the Tomcat service will peg the CPU at 100% when the clearspace.war file is dropped into the webapps folder and the Tomcat Windows service is restarted. It appears that the service is hung, but it eventually recovers and then begins to run normally.

Here's the sequence of events as it happened for me:

Stopped the Tomcat Windows service.
Backed up jiveHome directory and clearspace database
Removed the existing clearspace directory and clearspace.war file from the webapps folder.
Modified the WAR file by using the "EditWAR.jar" tool to modify my jiveHome directory.
Dropped the new WAR file into the webapps folder.
Restarted the Tomcat Windows service.
Confirmed that the clearspace folder under webapps is generated.

At this point, the CPU is pegged at 100% by the Tomcat service (tomcat5.exe). After several minutes, the utilization dropped and I was able to proceed with the upgrade process by pointing the browser to the Clearspace site.

After completing the upgrade, the instructions stated that I needed to restart the application. Since I had the service MMC up and Clearspace is the only app installed in this Tomcat installation, I just restarted the Tomcat service. Again, after restarting the service, the CPU was pegged at 100% for several minutes before it became usable again.

I have confirmed that the application causing Tomcat to peg the server is Clearspace because I have restarted the service several times successfully without the clearspace.war file deployed.

Currently at this point, while the app appears to have been upgraded successfully and configured properly, Clearspace now causes 100% CPU utilization upon any request of the web application, or whenever the application is reloaded from within the Tomcat Web Application Manager.

Does anyone have any ideas what would be causing this? Have I missed some critical configuration piece for my scenario?

Here's a summary of my setup:

- Windows 2003 Server Enterprise SP2

- Sun JVM 1.6.0_01-b06

- Clearspace 2.0 (version downloaded today)

- Apache Tomcat 5.5.26

I'd love to hear the simple and silly thing that I missed which is causing me so much pain.

Thanks,

Chad

Tags: 2.0, clearspace_windows, tomcat, clearspace

Jochen Kirchgessner

17 posts since
Feb 19, 2008

Apr 8, 2008 2:01 PM

in response to: casadyc

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Hi Chad,

I'd love to give you the reason for your problem but I cant. Even worse: same here. The environment:

- Windows XP SP2

- Sun JVM 1.6.0_01-b06

- Clearspace 2.0

- Apache Tomcat 5.5.26

- MySQL 5.0.51

After waiting for Tomcat bloating up to about 150 MB while using 100% CPU (takes about 2 to 3 minutes), everything runs well. This happened after I got the Clearspace Community Beta 6 war file and deployed it to Tomcat using a virgin jiveHome folder. Any explanation for possible mistakes during setup would be helpful.

Cheers,

Jochen

Beau Gordon

72 posts since
Aug 27, 2007

Apr 8, 2008 3:16 PM

in response to: Jochen Kirchgessner

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Me too.

Windows 2k3 server

Tomcat 6.0.14

Java 1.5_13

MySQL 5.0

It's taking substantially longer than a couple of minutes in my case, though this may be because I am running in a VM and not giving enough resources to clearspace.. This is just a clean install in a test environment.

casadyc

33 posts since
Apr 27, 2007

Apr 8, 2008 3:17 PM

in response to: Jochen Kirchgessner

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Jochen,

In your implementation, does Clearspace run fine after Tomcat gets done making full use of the CPU? I'm still having severe performance problems, even after it's been running for a couple of hours. Every hit to the web server causes the CPU on my server to spike to 100% for a few moments and the response to the client is ~7-10 seconds.

It's almost like the Tomcat server is having difficulty caching the Clearspace application, and it's reloading it upon every request.

Thanks for posting your response -- misery loves company!

Chad

Jochen Kirchgessner

17 posts since
Feb 19, 2008

Apr 8, 2008 3:32 PM

in response to: casadyc

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Chad,

I can confirm that after this "initialization" process, Clearspace runs fine on my system. Sorry to hear it's not like that for you. Further I have to correct myself, I'm running Tomcat 6.0.16 - sorry about the wrong information.

Jochen

seth

670 posts since
Feb 12, 2008

Apr 8, 2008 4:34 PM

in response to: Jochen Kirchgessner

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

If you're experiencing a brief period of high CPU usage right after you've installed the clearspace.war is probably due to Tomcat deploying it. That's expected behavior, but shouldn't last more than a minute or so on modern hardware (your milage may vary if you're using a VM).

- Seth -

casadyc

33 posts since
Apr 27, 2007

Apr 8, 2008 4:50 PM

in response to: seth

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Seth,

Thanks for the response. The high CPU usage lasts for quite some time, and continues with every request of the server. It works for a while (however very slowly), before Tomcat finally pukes on the screen and reports a 500 error to the user:

"java.lang.OutOfMemoryError: Java heap space"

The full exception is:

javax.servlet.ServletException: Filter execution threw an exception
	org.apache.struts2.dispatcher.ActionContextCleanUp.doFilter(ActionContextCleanU p.java:99)
	com.jivesoftware.community.web.filter.JiveCacheFilter.doFilter(JiveCacheFilter. java:204)
	com.jivesoftware.community.web.filter.JiveDynamicResponseHeaderFilter.doFilter( JiveDynamicResponseHeaderFilter.java:66)
	com.jivesoftware.community.web.filter.PresenceFilter.doFilterInternal(PresenceF ilter.java:134)
	org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilt er.java:75)
	org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingF ilterProxy.java:183)
	org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterP roxy.java:138)
	com.jivesoftware.community.web.filter.JiveCompressionFilter.doFilter(JiveCompre ssionFilter.java:103)
	com.jivesoftware.community.web.filter.SetRequestCharacterEncodingFilter.doFilte r(SetRequestCharacterEncodingFilter.java:73)
	org.directwebremoting.servlet.DwrWebContextFilter.doFilter(DwrWebContextFilter. java:91)
	com.jivesoftware.base.plugin.PluginFilter.doFilter(PluginFilter.java:75)
	org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChain Proxy.java:265)
	com.jivesoftware.community.aaa.JiveAuthenticationTranslationFilter.doFilter(Jiv eAuthenticationTranslationFilter.java:99)
	org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChain Proxy.java:275)
	com.jivesoftware.community.aaa.FeedsBasicProcessingFilter.doFilter(FeedsBasicPr ocessingFilter.java:146)
	org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChain Proxy.java:275)
	org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeP rocessingFilter.java:142)
	org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChain Proxy.java:275)
	org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter .java:271)
	org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChain Proxy.java:275)
	org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSess ionContextIntegrationFilter.java:249)
	org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChain Proxy.java:275)
	org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
	org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
	com.jivesoftware.community.web.filter.ApplicationStateFilter.doFilter(Applicati onStateFilter.java:112)
	com.jivesoftware.community.aaa.SecurityHolderCorruptionCheckFilter.doFilter(Sec urityHolderCorruptionCheckFilter.java:29)

I'm getting ready to try another JRE to see if that is my problem.  Do you have any suggestions?

Chad

seth

670 posts since
Feb 12, 2008

Apr 8, 2008 5:05 PM

in response to: casadyc

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Hi Chad,

Based on the error, you can try increasing the amount of memory available to Clearspace.

For our recommended memory configuration, take a look at our Recommended Environments doc.

In your case, try putting the following in your <tomcat install dir>\bin\startup.bat file (anywhere before the line that says call "%EXECUTABLE%" start):

set JAVA_OPTS=-Xms512m -Xmx1024m -XX:MaxPermSize=256m

- Seth -

Beau Gordon

72 posts since
Aug 27, 2007

Apr 9, 2008 8:27 AM

in response to: seth

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Seth,

The thing that is causing me some concern is that I'm having this problem using the exact same configuration which has worked just fine for 1.x versions of Clearspace. Usually with 1.x, restarting Tomcat spikes the CPU for 10-15 seconds. No problem. With 2.0, that spike is lasting for >30 minutes for me. In my case, it looks like once it gets through that spike, it works properly. I am using a VM, but I have the same Java memory configuration that you recommended.

Maybe this is a Windows+Tomcat thing, as that seems to be the commonality between the people on this thread.

Beau

seth

670 posts since
Feb 12, 2008

Apr 9, 2008 10:54 AM

in response to: Beau Gordon

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Hi Beau,

How much memory do you have allocated to Tomcat (and to your VM)?

Can you try running Clearspace outside of your VM and see what your performance is like?

- Seth -

Beau Gordon

72 posts since
Aug 27, 2007

Apr 9, 2008 11:07 AM

in response to: seth

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

My VM has 1.5 GB on a P4 3.0 Ghz machine. No other VMs are running.

My java options are set at 512 min/1024 max heap and 256 for the permsize, same as your recommendations.

I don't think this is a memory issue. Tomcat is using 182 MB right now and my DB is empty. I never get to the point where I am paging memory. It's just the CPU. It acts like there is something that it gets stuck on while Tomcat is starting up.

I appreciate your help so far, but I feel you need to address the reason why there is such a big difference between 1.x and 2.0. I'm seeing a 12000% increase in startup time between the two versions using the exact same Java and Tomcat settings. That seems like a red flag that should be investigated.

seth

670 posts since
Feb 12, 2008

Apr 9, 2008 11:46 AM

in response to: Beau Gordon

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

If you're not using Web Services, you might try disabling them. That should improve startup time somewhat. You can do this by setting -Djive.ws.disabled=true in your JVM options.

I can understand the inconvenience of having longer startup times (especially when you're doing migration and testing out settings), but under normal use cases I wouldn't expect you to have to start Clearspace that often.

- Seth -

seth

670 posts since
Feb 12, 2008

Apr 9, 2008 5:25 PM

in response to: Beau Gordon

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Hi Beau,

I've been doing a bit of research on this, and adding the -server JVM option has significantly improved startup performance. In some of the tests that I ran, adding -server reduced the startup time by a factor of 4.

Setting: -XX:UseConcMarkSweepGC -XX:UseParNewGC may help runtime performance as well.

- Seth -

casadyc

33 posts since
Apr 27, 2007

Apr 9, 2008 5:58 PM

in response to: seth

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Thanks for the research Seth!

Beau, the syntax for the JVM options below might be a little different depending upon your JRE and Tomcat version. For me, the syntax is "-XX:UseConcMarkSweepGC -XX:UseParNewGC" with a plus sign.

Also, I found an article about configuring JVM options for Tomcat here (section 3.2):

http://www.chemaxon.com/jchem/doc/admin/tomcat.html.

The author states that "One should not specify the '-server' or '-client' JVM option," because the server .dll is placed under the 'server' directory in the JRE directory. After looking at my setup, I saw that Tomcat is using the jvm.dll file from the "C:\Program Files\Java\jre1.6.0_05\bin\client\jvm.dll" folder, and I don't have a "server" folder. When I tried to use the "-server" option, Tomcat wouldn't start at all. So I'm thinking I've got some configuratin' to do.

I'm going to have to do more research to see if I've got the wrong JVM setup and if changing that makes a difference.

Chad

Bruce Ritchie

2,286 posts since
Jun 28, 2001

Apr 9, 2008 7:15 PM

in response to: casadyc

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Chad,

I would suggest using the JDK, not the JRE and explicitly set the -server flag to force the JKD to use that VM. Depending on the JDK and OS the JDK may or may not use the server JVM by default - I think the JDK on Windows XP/Vista would use the client JVM unless otherwise told to use the server JVM.

Regards,

Bruce Ritchie

casadyc

33 posts since
Apr 27, 2007

Apr 9, 2008 8:32 AM

in response to: seth

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Wow! Clearspace 2.0 is a memory hog! I was previously running Clearspace 1.10 on a slimmed down Windows 2003 VM with basically nothing but Tomcat/Clearspace and 256MB of RAM. Memory rarely peaked over 200MB, and the CPU averaged a comfortable 5-10%. With the new version, I had to change my VM to 1GB RAM just to get it to be usable, and the CPU still consistantly jumps to 50% and higher for regular page requests.

I love the app, but in my opinion, the increase in the hardware requirements should be proportional to the increase in features. It almost makes me want to start comparing Clearspace's hardware requirements to a certain collaboration tool made by our friends in Redmond!

Chad

seth

670 posts since
Feb 12, 2008

Apr 9, 2008 10:49 AM

in response to: casadyc

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Hi Chad,

I have a Clearspace 2.0 installation that idles at around 150 MB, and a 1.10.5 install that idles around 100MB. These are both test installs without much data in them, so those numbers are probably good estimates of the minimum memory requirements.

If you're running Clearspace in a virtual machine, you are definitely going to benefit from more memory. Remember that if a Windows virtual machine has low memory, it's going to start paging memory out just like a real Windows machine will (swap memory on virtual machines is a real performance killer).

If your (physical) server supports it, you should enable VM hardware acceleration (in the BIOS). That will usually give you a 10-20% performance boost. Also, if you have your database server on the same VM (or on a VM at all), you should move that to a real server as well.

That "certain collaboration from Redmond" lists a 2.5 GHz processor with 1GB of RAM as for it's minimums, by the way

- Seth -

casadyc

33 posts since
Apr 27, 2007

Apr 9, 2008 1:32 PM

in response to: seth

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Here it is in a nutshell.

The virtual machine that I had set up ran Clearspace 1.10 perfectly prior to my upgrade to Clearspace 2.0, and with only 256MB of RAM system RAM it rarely went over 200MB. The only difference between then and now, is that I upgraded to Clearspace 2.0, and now the memory requirements are far higher than before.

I expect an increase in the amount of memory required for a full version upgrade, but this seems far more than the norm.

jbaldwin@rileykids.org

20 posts since
Mar 15, 2007

Apr 9, 2008 10:03 AM

in response to: casadyc

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

Yep - same here. Except, that the CPU nevers settles down (left Tomcat running for at least 30 minutes before I killed the java.exe in the task manager). My experience with the 1.x versions was that a launch would cause a CPU spike that would settle down after a few minutes. This version never gives up its impact on the processing environment.

BTW: I boosted the Tomcat memory settings per the above suggestion: set JAVA_OPTS=-Xms512m -Xmx1024m -XX:MaxPermSize=256m

The box I am using for Clearspace is running Windows Server 2003, Celeron processor, 4gb RAM. The server also runs Openfire and MS WSUS.

casadyc

33 posts since
Apr 27, 2007

Apr 9, 2008 10:00 AM

in response to: jbaldwin@rileykids.org

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

That was the same scenario for me until I boosted the memory on my VM. It's running fine now, but I was really wanting a much smaller footprint.

Chad

jbaldwin@rileykids.org

20 posts since
Mar 15, 2007

Apr 10, 2008 7:10 AM

in response to: casadyc

Re: Clearspace 2.0, Tomcat 5.5 service hangs on Windows

I'm not that technically inclined - nor, do I know much about Java and VM's. Should I boost the memory using the following spec:

JAVA_OPTS=-Xms512m -Xmx1024m -XX:MaxPermSize=256m

If so, what do you recommend setting it to?

Clearspace is not working now. I am considering going back to 1.10.5. That at least worked.

thanks!