Web ClayStation

Web ClayStation is a Web application, a Web version of the ClayStation client which provides the most important features of a Clay client through a standard Web-browser. It consists of:

• Any standard Web-server that supports FastCGI with an installed FastCGI module.
• A Web Clay FastCGI module that runs either on the same machine as the Web-server or on a remote machine accessible by the Web-server. Web Clay connects to ClayCentral.
• The various files needed by both the Web-server and Web Clay, including HTML files, JavaScript code, style sheets and resource files.

Web ClayStation is a complementary solution and technology to the desktop version of Digital-Clay clients, offering the advantages of a Web client for those that need it. See the article Desktop vs. Web Clients for an in-depth discussion on the advantages and disadvantages of both approaches, and the specific uses, scenarios and goals that apply to each solution.

The Web ClayStation is meant to be used by employees and managers but not administrators. As such, it provides all the Digital-Clay tools necessary for viewing and changing data, but no administration, system and profile customization tools.

The following Digital-Clay tools and features are missing or limited and should be accessed using ClayStudio/ClayStation desktop clients:

• All administration tools for configuring ClayCentral, connection management and system statistics.
• All database customization tools for changing custom tables, fields, field properties and permissions.
• All tools for configuring Automation rules.
• All profile customization tools for changing toolbars, dashboards and layouts.
• Some intensively interactive components such as the Gantt chart and Calendar.
• Charts, Gauges and Graphs are static pictures instead of being interactive with context-sensitive mouse clicks.
• No Clay auto-updates. Although the users receive their updates automatically when the server is updated, the server module itself must be updated manually.

This section is for administrators and integrators wishing to get a deeper understanding of the design and architecture of the Web client for troubleshooting purposes.

The diagram on the right presents the layered architecture stack of the Digital-Clay platform and the exact positioning of the Web Clay module and server. Note that:

• The Web Server communicates with the Wec Clay module using a FastCGI protocol similar to CGI only with more capabilities and the ability to connect remotely via TCP.

• The Web Clay module connects to ClayCentral using the same communications layer as a Clay desktop client application.

• Applications may call Clay REST API via Web Clay.

FastCGI is a standard and widespread extension of CGI available since 1996 that corrects some issues and limitations of CGI:

• FastCGI modules stay in memory after they are loaded, thus eliminating the heavy and slow overhead of spawning a process for every request.

• Since FastCGI stays in memory between requests, there is no overhead of spawning new connections, loading variables, settings and data needed for every request. I.e. FastCGI allows for more possibilities and faster caching of data and connections.

• FastCGI modules can be communicated with via TCP, allowing the modules to run on separate machines.

• Like CGI, FastCGI modules can be written in any language and can therefore make use of fast, optimized and pre-compiled code.

Which all adds up to greatly increased performance. In addition, the FastCGI specifications allow for the following communication enhancements:

• Automatic spawning and handling of multiple FastCGI processes for handling of multiple requests.
• Re-using the same FastCGI connection between requests.
• Multiplexing requests from different clients on the same connection.
• Since FastCGI modules remain in memory, they can employ multi-threading to deal with the above scenarios and handle multiple requests even using a single process.

In reality, different Web-servers implemented varying levels of the specifications, so although the Clay FastCGI module supports all of the above mechanisms, its capabilities depend on the specific Web-server that was installed.

Note that technically, it is a separate web module that enables FastCGI on the Web-server therefore there may be several available options for one Web-server (e.g. Apache), or it may even be installed automatically with the Web-server (e.g. IIS7). See the FastCGI Site for more details.

This is the simplest scenario most suitable for a company that hosts everything internally and that usually doesn't have too many concurrent requests or heavy CPU usage on the server:

• The Web-server communicates via Pipes which are slightly faster than TCP.
• All server applications are on one machine.
• Only one FastCGI process is spawned by the Web-server for handling all requests.

If the Web-server supports FastCGI multiplexing or multiple Pipe connections, then this configuration can handle concurrent requests. However, some Web-servers in practice (including IIS and Apache) can only only handle one request/Pipe connection per process therefore multiple requests will be queued and handled one at a time in this scenario.

This scenario is for better performance as well as for handling concurrent requests on any Web-server:

• ClayCentral/database is running on a separate machine so that it gets more resources to handle requests as well as other clients.
• The Web-server spawns several FastCGI processes and distributes the requests between them.
• The FastCGI modules automatically spawn their own singleton server process that handles all requests using multithreading, a single global cache and connection pool.

This scenario demonstrates a case where the Web-server is running on a separate (hosted? Unix?) server and therefore must communicate with Clay remotely.

• The Web Clay module is run automatically at startup on a separate machine on the company network. It must be accessible via the internet.
• Web-server to Web Clay communication is via TCP/IP.
• ClayCentral is running on an internal company server that also serves the desktop clients.

Note that some Web-servers like Microsoft IIS do NOT support remote FastCGI via TCP/IP!

The quick overview to installing Web ClayStation is as follows:

1. Make sure you have a Web-server installed that supports FastCGI.
2. Install the FastCGI module for your Web-server. Note that the IIS FastCGI modules only work on Windows Server (2003+).
3. Install ClayCentral if you have not already done so. ClayCentral should ideally run on its own machine if many concurrent users or heavy requests are expected.
4. Install Web ClayStation on the appropriate machine. The 'All Users' setting in the install must be left on. See above regarding sample machine configurations.
5. Copy the 'clayres' folder (installed with Web ClayStation) to the root folder of the web server. If it is not in the root folder, then see below for Clay Configuration settings.
6. (Optional) Copy any Clay Document Templates you want to make available to web-users under the C:\DigitalClay\Templates\ directory. This location is configurable.
7. Configure the Web-server FastCGI so that it runs/connects to the Web Clay module. See Web Server Configuration below for more details.
8. (Optional) Configure the Web Clay module if necessary. See Clay Configuration below.

• The Web Clay module does not update itself automatically as with other Digital-Clay clients and must be updated manually along with its resource files and the 'clayres' directory.

• Obviously, this must only be done once on the server and all users will then receive their updates automatically.

• If Web ClayStation is being used, automatic updates on ClayCentral should be disabled otherwise it may update itself and the Web Clay module will then not be able to connect until it is updated.

• All claycgi.exe processes must be killed before an update to avoid locked files. With some Web-servers, this will probably necessitate taking down the Web-server itself.

Configuring the Web-server and its FastCGI module is very specific to the server in question as well as the configuration scenario. The typical approach is to add a (virtual or physical) file that is redirected to the claycgi.exe executable. This is usually done by mapping a specific file extension (e.g. .dcw) to the claycgi.exe executable using the FastCGI or Web-server configuration file or tool. Other key points that may arise are:

Permissions:

• (Handled by default built-in Apache installation). The web alias for Apache (in our example we call it 'clay.dcw') and claycgi.exe need to have executable permissions. Check both OS/file system permissions such as NTFS as well as Web-server administration tools.

• (IIS) With some servers like IIS6, the .dcw extension needs to be mapped to the FastCGI executable separately. (e.g. for IIS6 go to the folder properties/Application Configuration and map .dcw to 'C:\WINDOWS\system32\inetsrv\fcgiext.dll').

• (Handled by Clay installer unless folder moved). Read-permissions need to be given to the 'clayres' folder and its subfolders.

• See Automation/WebClayStation Create Document Errors regarding Microsoft Office issues.

• (General guideline:) On some systems, permissions may need to be given to the claycgi.exe process itself so that it can access all the resources it needs, keeping in mind that it is usually running with the privileges of either 'System'(Apache), 'Local Service' or 'Network Service'(IIS6), or 'IIS AppPool\DefaultAppPool' (IIS7) on Windows. Check what user the claycgi.exe process is running under and give that user (or you may simply add the special user 'Everyone') the following permissions:

• (Usually not needed for the System user and in newer Windows versions) Read/Write permissions to the 'Temp' and 'Local Settings\Application Data' folders under the user's Windows profile folder (e.g. in Documents and Settings). Note that due to Windows quirks, for 'Network Service' this is sometimes the 'DefaultUser' profile folder, and for 'System', this is sometimes under '\Windows\system32\config\systemprofile'. The temp folder is usually \Windows\Temp but this may change as well.

• (handled by the installer) Read permissions for the installed 'html' and 'lang' folders in the Digital-Clay installed directory.

• (handled by the installer) Write permissions to the ClayCGI.log file

• (handled by the installer) The document template folders in the c:\DigitalClay folder

• (Usually only needed for the IIS Windows user and not the System user, but worth a check) In order to allow the claycgi.exe process to run Microsoft Excel or Word (for exports and templates), it must be given permissions to those applications as well: Go to Control Panel/Administration/Component Services/Component Services/Computers/My Computer/DCOM Config, find the application (e.g. 'Microsoft Excel Application' or 'Microsoft Word Document'), open its Properties/Security and give the appropriate Windows user (System/etc) access to Launch, Access and Configure this application locally.

• (To remove MSIserver event log errors only) To avoid DCOM errors in the Event Log related to MSIserver, you also have to give the appropriate user (e.g. Network Service) component permissions as well. Use the above instructions and give the same permissions to the component: {000C101C-0000-0000-C000-000000000046}

Configuration settings:

• The maximum amount of processes that are spawned OR the IP of the remote FastCGI module.

• The timeout settings for requests and connections so that long requests don't time-out.

• The timeout for idle processes or the maximum amount of requests that each process can handle should be maximized. I.e. Ideally, the Web-server should not automatically kill processes when they aren't being used or when they've been used too often - this may disconnect users for no reason.

• The maximum amount of requests to queue in case the server is overloaded and users are waiting too long.

This is the latest and fastest official FASTCGI module that is included in Apache 2.4+. It supports only TCP/Remote connections (which are faster).

Sample/recommended configuration of httpd.conf file with full comments/explanations (Windows)

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_fcgi_module modules/mod_proxy_fcgi.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
<Proxy "balancer://clay">
    BalancerMember "fcgi://127.0.0.1:2010/" retry=0 timeout=180 acquire=30000 connectiontimeout=30
	ProxySet maxattempts=30 forcerecovery=On timeout=30
</Proxy>
<IfModule mod_proxy_fcgi.c>
  <Location /clay.dcw>
   SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
   ProxyPass balancer://clay
  </Location>
  <Location /claysso.dcw>
   SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
   ProxyPass balancer://clay
  </Location>
</IfModule>

Explanation for ProxyPass or BalancerMember settings:

• retry (seconds): How long to wait before re-using failed worker thread. If Clay is unresponsive we don't want to disable a worker thread, so retry imemdiately. Default is 60
• timeout (seconds): How long to wait for Clay to respond to a request before timing out. Adjust as necessary. Default is 60
• connectiontimeout (seconds): How long to wait for connections to establish with Clay.
• acquire (milliseconds): Maximum time to wait for a free connection in the connection pool. Returns busy error if no free connections by default.
• (optional for testing) max: To limit amount of concurrent requests. Default = amount of thread workers. If max=1 it will wait til response before sending next one.
• Do NOT use enablereuse=On: It doesn't work properly even though it may increase performance a bit.

Proxy balancer settings set globally via ProxySet:

• maxattempts: How many times to retry. This is critical when Clay is too busy to respond.
• orcerecovery=On: Force recovery if all failed, and ignore retry time. Not needed because retry=0 plus default is On.
• timeout (seconds): How long to wait for worker thread to be available in balancer.

See mod_proxy_fcgi and mod_proxy for configuration options.

To configure the claycgi module itself running independently in TCP mode as a Windows service, see Clay Configuration below.

mod_fcgid is a fastcgi module developed by the Apache project team but currently it does not yet support TCP/remote mode and it performs slower than the other modules. It works with Apache 2.4+.

Pipe/local mode with four spawned processes:

LoadModule fcgid_module modules/mod_fcgid.so
<IfModule fcgid_module>
  FcgidMaxProcesses 4
  FcgidProcessLifeTime 0
  FcgidMaxRequestsPerProcess 0
  FcgidIdleTimeout 0
  FcgidIOTimeout 60
  FcgidConnectTimeout 60
  FcgidInitialEnv CLAYCGI_MODE Pipe
  FcgidInitialEnv CLAYCGI_MULTIPROCESS 1
  <Location /clay.dcw>
     SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
     SetHandler fcgid-script
     FcgidWrapper "C:/Program Files/Digital-Clay/Web/claycgi.exe" .dcw virtual
     Options +ExecCGI
     Order allow,deny
     Allow from all
  </Location>
</IfModule>

See mod_fcgid Reference.

IIS7 and up includes the FastCGI module as part of the Web Server and must be configured using the built-in configuration tools. See the following links for general guidelines:

• http://www.coastrd.com/cgioniis7
• http://www.iis.net/ConfigReference/system.webServer/fastCgi

If the FastCGI Settings module does not appear, you must install the Administration Pack for IIS7.

The general steps are:

1. In “Handler Mappings”, add a “Module Mapping” from *.dcw to the claycgi.exe
2. In “FastCGI Settings”, configure the claycgi.exe module:
   • Add environment variables as needed, minimally add CLAYCGI_MODE=Pipe and CLAYCGI_MULTIPROCESS=1
   • MaxInstances=4
   • InstanceMaxRequests=10000000
   • IdleTimeout=604800
   • QueueLength=100
   • ActivityTimeout=60
   • RequestTimeout=90
3. Set the permissions as usual (see Web Server Configuration)

A Web Clay FastCGI process is configured using either command line parameters, or environment variables, or both. Command-line settings override environment variables but can use an environment variable to temporarily set a setting for the WebClay service, such as the loglevel. These two options are provided to accommodate different Web-servers' capabilities.

Command-line variables can be added to the WebClay service by editing the registry value for the 'Command' entry under:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\WebClayStation\Parameters

Any configuration setting listed below can be converted to an environment variable by prepending CLAYCGI_ and converting it to uppercase (e.g. -mode becomes CLAYCGI_MODE).

-mode (Required)
Values: Either 'TCP' or 'Pipe'. This defines what method the Web-server will use to communicate to the Web Clay FastCGI module.
If this parameter is missing, the process will exit.

-multiprocess
Defines whether this module is running in a multiprocess scenario and is one of many processes and should therefore use a singleton server process for handling requests. See Sample Configurations above for an explanation of this scenario.
Default: (off)

-port
The port to listen on if using TCP communications only.
Default: 2010

-resurl
The URL path/location on the web site where the Clay resource files are located.
Default: /clayres/

-addonurl
The URL path/location on the web site where addon html files are located. This is only needed if addon html files are used inside layouts, etc. that use the default addon location parameter in their URL (see Add-Ons/External GUI).
Default: /clayaddons/

-clayserver
The IP of the ClayCentral machine.
Default: 127.0.0.1

-clayport
The port used by ClayCentral.
Default: 1007

-deflang
The default language to use (e.g. 'EN') for the login screen and as a default for all users (note that each user can override this setting with his own language in the options).
Default: The default language set in the Windows Regional Settings.

-reqtimeout
The maximum amount of seconds to wait for responses from ClayCentral before timing out and returning an error to the user.
Default: 50 seconds

-conntimeout
The maximum amount of idle minutes to allow a user before he is disconnected from ClayCentral. This is necessary in case the user forgets to logout before he closes the Web-browser.
Default: 30 minutes

-maxthreads
The maximum amount of internal threads to spawn for handling multiple concurrent requests. Keep in mind that too many threads may impede performance but many threads may be simply waiting for responses from ClayCentral.
Default: 10

-ssokey
This is mandatory if using Windows Single-Sign On as the authentication method for WebClay clients (web browsers). This is a secret key that is only shared between ClayCentral and WebClay. The same value must be configured both in ClayCentral settings ('WebClay SSO Key'), and in WebClay using the ssokey setting. It can be any value the admin chooses.

-webips
A security setting for limiting which IPs can connect to this FastCGI process via TCP. This is to block any connection/hacking attempts from outside the web server. The value is a semi-colon separated list of IPs.
Default: (Empty - i.e. all can connect)

The users logging in through the Web ClayStation will receive either default profiles, or the profiles configured and saved using Profile Management tools in ClayStudio, or the profile the user himself saved in a desktop client using 'Save My Profile'.

For more details on customizing a user's toolbar, homepage, settings and layouts, see Profile Management.

Errors while running the Clay WebStation must be differentiated as follows:

This means that the expected response is received without any errors, but the response was exceedingly slow. As explained below, this usually should be distinguished from time-out errors which may be due to bugs or non-performance related problems. With performance issues, the bottleneck must be found which may be due to a variety of sources including: The client machine or operating system, the internet connection, the bandwidth, the web server, faulty network cards, routers or cables, the server hardware, slow network software such as firewalls, VPNs and QOS/packet shapers, and finally, an overloaded FastCGI module, ClayCentral or database. Each of these can be ruled out by a process of elimination or by using standard network and hardware troubleshooting techniques.

An application error should be differentiated from a time-out error: A time-out response usually takes a long time (40 seconds or more) and may be mistaken for a performance issue if the user doesn't wait for the time-out to occur (e.g. he hits refresh before the time-out error appears); An application error is usually received within a few seconds.

Both of these errors may come from different sources. It is important to understand each the following elements so that the proper source is troubleshooted and the appropriate administrator called. The elements that may return errors are: Browser→Web Server→FastCGI→ClayCGI→ClayCentral:

• If the error/time-out is due to a connectivity issue between the Browser and the Web Server, or due to an unresponsive Web Server, you will get a browser-dependent error. This may be due to a connectivity/networking problem, or it may be a performance issue with the Web Server or network (see above).

• If the error occurs between the Web Server and the FastCGI module, either because the FastCGI module is unresponsive or misconfigured, the error usually defines itself as a Server error with the name of the Web Server (e.g. 'IIS') in the error text, or as an “Internal Server Error”. To further clarify the source and cause of this error, the web server log should be studied. A time-out is very rare in this scenario since the FastCGI usually handles and responds with its own time-out errors (see below).

• If the error occurs between the FastCGI module and the ClayCGI module, either due to a misconfigured ClayCGI, or a ClayCGI communications error, then you should get a “FastCGI error” in IIS, or an “Internal Server Error” in Apache. To further clarify the source and cause of this error, the web server log should be studied as well as the ClayCGI log. The log files should also point out where a time-out is occuring since the problem can either be due to an unreachable ClayCGI (configuration issue), or due to the ClayCGI not responding in time due to an unexpected communications error (bug).

• If the ClayCGI module encounters a non-communications error in its code, or ClayCentral sends back an error, or the user ran an illegal Clay action, the error will appear in the browser with a large red icon. Even if the error is unexpected but not a bug in the communications, the user will receive a red ClayCGI time-out error and the ClayCGI log file should be studied. Note that not all Clay errors are bugs, and only bugs or unexpected errors appear in the log file.

Summary: A time-out error only appears if the user waits for it, and it may be either due to bugs, OR performance problems. An application error is almost always due to misconfigurations or bugs. Slow responses are almost never bugs.

When checking the web-server or ClayCGI log files (\Program Files\Digital-Clay\Web\ClayCGI.log), always compare the timestamp of the errors to the time the problem occured to ensure that the log entries are relevant.

Due to the fact that every Web-server and FastCGI module is different, troubleshooting the FastCGI configuration is usually a system-dependent matter. But try the following checklist:

• Double-check everything in the configuration file including exact file locations (pathnames), filenames, and port numbers.
• Make sure there is no internet protection, firewall or antivirus software that may be blocking the claycgi.exe processes.
• Double-check the permissions for all resource files and locations, as listed above.
• Check the Web Server log and ClayCGI.log files for errors.
• It sometimes helps to run the claycgi.exe file once manually with administrator privileges so that it can register itself properly before restarting the Web-Server.

The workload of a Web Claystation request is typically distributed over 4 levels and each one of these levels (or the networking speed between them) may become a bottleneck:

1. The ClayCentral database. Optimization options: Move to SQL Server, upgrade hardware such as memory speed, memory size, FSB, hard drive speed, CPU, or with heavy concurrent usage, move SQL Server to its own dedicated machine.
2. ClayCentral: Dedicated machine, better memory, CPU and FSB speed, Maximum Threads Setting.
3. ClayCGI: Dedicated machine, better memory, CPU and FSB speed, maximum processes/threads settings (see below). As explained above, ClayCGI takes advantage of all the benefits of FastCGI including caching, open connections, multithreading and multiplexing, thus resulting in top speeds surpassing other web technologies.
4. The Client Web Browser for drawing and populating the GUI elements. Client hardware can improve performance as well, but more importantly, since Claystation is Javascript intensive, a faster Browser like Opera or Chrome or even Firefox can make a big difference.

Obviously, the better the underlying hardware such as CPU, memory speed, and FSB speed, the better the response times from ClayCGI. It is also important to have fast networking speed between ClayCGI and ClayCentral since large amounts of data are transferred between them. For this reason, it is not a good idea to host the ClayCGI externally and have it connect to a ClayCentral on a LAN, unless performance is not a concern.

On systems with many concurrent requests, increased performance can be achieved by separating ClayCGI and ClayCentral to two machines, and possibly also separating the Web Server and ClayCGI to different machines (using TCP FastCGI comms with Apache for example), as long as the network between these machines is sufficiently fast.

Besides hardware and machine topology, an important factor for performance is the amount of processes and threads. Note that there are three elements that define the amount of concurrent requests that can be handled:

1. On IIS and other Web Servers that manage one process per request (usually when using Pipes for FastCGI comms - see above), the first bottleneck is the Web Server configuration setting that defines the maximum amount of processes to launch. For example on IIS, this is the MaxInstances setting.
2. Internally, ClayCGI spawns up to 10 threads to handle concurrent requests by default. To adjust this, use the -maxthreads setting.
3. For all requests that must be forwarded to ClayCentral, ClayCentral's internal threads may become the bottleneck in some rare cases. Adjust this using the ClayCentral Maximum Threads setting in ClayCentral Settings.

The ideal amount of ClayCGI processes/threads obviously depends on too many variables, but a recommended starting point is as follows:

Minimum 3, maximum 2 more than the amount of CPUs. So, for example, if you have a Quad-CPU system, and maximum concurrency is important, set the processes/threads setting to 6.

For assistance with fine-tuning these settings, some very basic statistics are collected and placed in the C:\DigitalClay\ClayCGI\ folder as .csv files. These files may be used to monitor average response times after adjustments are made. Contained in these files (one file per month) are the following columns for analysis:

• Date: The statistics are saved once per hour. Each row is thus a total for the hour displayed in the Date column.
• Activities: The total amount of requests that were handled.
• Activity Duration: The total amount of time spent actually handling requests inside ClayCGI excluding the time spent waiting for ClayCentral. Note that these times are per thread, so, for example, if two threads are actively running concurrently for 1 minute, the time displayed will be 2 minutes.
• Central Duration: The total amount of time spent waiting for responses from ClayCentral. As with Activity Duration, this is per thread. This number is separated as an indicator of how busy the ClayCGI is. When wait times increase it may mean that ClayCentral is the bottleneck or that ClayCGI could run more threads since they spend more time waiting than processing data.

To get the total response time, calculate Activity Duration + Central Duration. To get the average response time, calculate Total / Activities. To test performance for example, make adjustments, and compare the average response times before and after.