ClayCentral Service Stopped

[miplatt]

So far our administrator has not found anything unusual when checking the server.

Do you have other suggestions?

Every day, at least once, Central falls and we have to start the service again manually. On some occasions, it hangs and does not respond so we have to kill the program and start it again. This is not a feasible way to work.

[ZevToledano]

This is a strange mystery that needs investigation. Are you still getting out of memory errors in the Central log right before it fails?

If so, then one thing I can think of is that there may be one request that one of your users runs that demands a lot more memory, and whenever this is run, ClayCentral fails. I would systematically go through all of the daily activities that your users run on Clay, and try to purposely make ClayCentral fail.

Come to think of it, if the above is correct, then ClayCentral should be failing during work hours and your users should be getting disconnected. Is this the case? If yes, then next time it happens ask all your users what they were running when it happened.

Sorry, but this is all I can think of at the moment. Except for some double checking with the administration of the machine:
- For example, did he check that the paging file is OK?
- Is there plenty of RAM left over throughout the day? Perhaps there is another process on the server that runs once a day that takes up all the RAM and this causes ClayCentral to run out... Try to run a monitor on the RAM that sends a warning when it gets low.
- When the ClayCentral service fails, have Windows send a msg to someone in the office who can check right away what else is running and the status of the RAM.

[miplatt]

Our administrator has been running tests and said that paging file is ok. So far no major clues otherwise.

Our administrator changed the settings of the Digital Clay service so that it automatically restarts itself when it crashes. In addition, he installed a program called Health monitor that monitors the condition of the service and of the system memory. It sent me an email yesterday afternoon at 1:57 PM as an email alert of a problem, though Central was not running throughout the day (as you can see in the log).

The restart after 11 PM was done manually after noticing that Central was down.

Most recently, as you can see in the log, Central stopped around 8:09 this morning and started again around 8:10. One user said he tried to sign onto clay at that point and received the message that he was not able to connect. I am thinking that it probably restarted on its own as a result of our administrator's change to the settings but I am not positive.

[miplatt]

As we were not able to logon anyway, I killed Central and restarted it.

Then only I logged on. I was able to use it normally for a few minutes.

Then tried to run a simple search in a dashboard filter and it just hangs and does not list any results. (When I put the mouse over the search results box it just shows the hourglass)

RAM on server appears to be ok.

[ZevToledano]

I am currently testing a theory. Please follow all of these instructions (on the ClayCentral machine):
- Rename ClayCentral.log so that we start with a new log file.
- Run regedit, go to hkeyLocalMachine\Software\DigitalClay\Central
- If it doesn't exist, add a new String value 'LogLevel' and set its value to 4
- All kinds of new things will appear in the log file, but if you see mail errors (check in a few hours if necessary), send me the new log file please.
- Tomorrow, delete the LogLevel entry from the registry, and rename/delete the ClayCentral.log file so you can start with a fresh log file again.

[miplatt]

Kill and restart Central service also?

[ZevToledano]

No.

[miplatt]

Ok, I was the only user connected and things seemed to be functioning normally (not according to the log but at least as far as I saw).

Then just now another user trying to connect. It said he has been disconnected from the server.

A few seconds later I also got the message that I am now disconnected from the server.

See the attached log.

[miplatt]

Now both of us were able to reconnect successfully.

[ZevToledano]

OK I think we have a solution. My theory is based on the fact that you have tons of mail errors in your log and I was wondering if there is a connection. I found a resource leak that happens when there is a mail error and fixed it.

Keep in mind that this is a resource leak which is why we were getting 'memory errors' even though the RAM seemed OK.

There is a new version out and up in a few minutes. I suggest you upgrade and let me know if the problem doesn't go away after the upgrade.

You also want to check why you are getting so many mail errors. Check the ClayCentral settings. The errors may either be from:
- The Mail tab: I doubt you are using mail connectivity so these settings should be empty.
- The Data Injection tab: If you are using mail injection then this should have mail settings configured, but they are probably badly configured which would cause the errors.