How-To: High Availability Setup for Orchestrator

How-To: High Availability Setup for Orchestrator

This Article describes how to setup an Orchestrator in a high availability scenario.


Prerequisites:

SSL Certificates / Load Balancer

  • Certificate must be issued to orchestrator DomainName which is stored as an alias in the LoadBalancer
  • The Orchestrator FQDN must be the subject name, the orchestrator nodes should be in the certificate as subject alternative name
  • All Robots are configured to connect to the Loadbalancer FQDN
  • The certificate must be installed on all orchestrator nodes
  • (The Orchestrator should close port 443 in general and just open it for the Loadbalancer)
  • The loadbalancer should be configured to use rond robin with a predictive setting should be used. NO session stickiness or persistance must be configured.

Database configuration

  • Recommended is to use an already established, high available sql server cluster. If it is not available, it needs to be created or the database gets the single point of failure.
  • If the database is not provided before the installation db_creator rights are needed for the user installing the orchestrator
  • db_owner rights should be given either to the application pool user or the SQL-User which is used to connect to the database

Shared Drive

  • Fileshare accessible from all orchestrator nodes for the application pool user
  • In most companies Windows/CIFS/SMB shares are already available. They are needed as a central place where the Orchestrators store the process packages and activities.
  • Those SMB shares should be high available too as they get the single point of failure if not.

Redis

  • If redis as master/slave is installed on an orchestrator node then a minimum of 8 GB RAM is needed for the installation
  • The REDIS master node should be a independent server as to not interfere with patching of the orchestrator nodes
  • The REDIS master should have a minimum of 4 GB RAM

Setup:

Setup Redis - Windows - Ports needed 6379 and 26379, for the example the master IP is 172.28.4.117

  1. Download redis for windows either here or find it in the dropbox under Dropbox (Roboyo)\02_IPA Implementation\UiPath\UiPath Server and the most recent version there.
  2. Copy the files on all nodes (only one master is possible, recommended are 3 nodes, always needs to be an odd number)
  3. On all nodes the redis.server.conf and the redis.sentinel.conf needs to be adjusted as follows:
    1. Put the local IP adress and port in the config under - 
      #ip of the local server
      bind 172.28.4.117
      port 6379
    2. If the node is the master no further configuration is required
    3. if the node is a slave the following changes need to be made
      #if the local server is slave, specify masterip and port
      slaveof 172.28.4.117 6379
    4. If a password has to be set then the following changes need to be made on all nodes
      masterauth 1234567
      Requiredpass - not sure!
  4. On all nodes the redis.sentinel.config needs to be aduste as follows
    1. Local IP and port 
      bind 172.28.4.117
      port 26379
    2. Set the master IP and port on all nodes
      #ip of the master server
      sentinel monitor redis-cluster 172.28.4.117 6379 2 (not sure what the 2 is!)
    3. If a password is required change the config accordningly on all nodes
      sentinel auth-pass redis-cluster <master-password>
  5. After this the configuration is complete install.cmd can be used to first install redis on the master and then on each subsequent node.
  6. Two services are installed:
    1. redis-server
    2. redis-sentinel
  7. After both services are installed the redis-cli.exe can be used to check the functionality of the redis cluster
    1. Usage to connect to the master node: redis-cli.exe -h 172.28.4.115 -p 6379 
    2. After the connection is established the command - info - can be used and among others the following stats should be displayed

    3. You can also check in the redis folder in the directory "state" and log file (need to look up again!)

Setup primary Orchestrator node

  1. Make sure that a accessible database and an accessible fileshare for all orchestrator nodes is configured
  2. Be aware that the connection to the database, in case of windows authentication, will be done with the user running the installtion. 
  3. Install the orchestrator normally but don't open the website.
  4. Open a PowerShell as Administrator and go to - C:\Program Files (x86)\UiPath\Orchestrator\Tools and execute the following command
    1. .\Configure-PlatformNode.ps1 -mode ConfigurePrimary -websiteName "UiPath Orchestrator" -nugetPackagesPath \\fileshare\NuGetPackages -redisServer 172.28.4.117 -redisPassword 1234 
      Note: The password is always required by the powershell command. If no password was set the web.config needs to be edited afterwards
    2. After the powershell script ran successfully a message "Configuration successful" is displayed and the configuration string for the other nodes is displayed - this string should be saved
      Example:
      & ".\Configure-PlatformNode.ps1" `
      -mode 'ConfigureSecondary' `
      -websiteName 'UiPath Orchestrator' `
      -nugetPackagesPath '\\fileshare\NugetPackages' `
      -nugetActivitiesPath '\\fileshare\NugetPackages\Activities\' `
      -nugetPackagesApiKey 'EE8B9EF0-DC7F-4AF6-8693-E48EEAC04246' `
      -nugetActivitiesApiKey 'EE8B9EF0-DC7F-4AF6-8693-E48EEAC04246' `
      -encryptionKey 'nHEtV1IbsB/Z5xvpIvkKNkjUWuKsK+2tLWPNgSvgwaE=' `
      -machineKeyDecryptionAlgorythm 'AES' `
      -machineKeyDecryptionKey 'B948A5247C472BA5D71B678ECEC395D861F3449CA85D31E0CF44D9A0ED10956C' `
      -machineKeyValidationAlgorythm 'SHA1' `
      -machineKeyValidationKey 'AC56F985C38FFFC4C0B901049EE78878F63F2B89A3A33C2253A2BDA2918C5323364371A22E8009B392B6839C138DC9E600D77F8F742B28026EF705BF9DF479F6' `
      -redisServer '172.28.4.117' `
      -redisPort 6379
    3. the web.config needs to be changed to reflect the different redis servers as shown below
      <add key="LoadBalancer.Redis.ConnectionString" value="node1:6379,node2:6379,node3:6379,password=bvbvbbv" />
      <add key="LoadBalancer.Redis.ConnectionString" value="172.28.4.115:6379,172.24.8.1166379,172.24.8.117:6379" />
      The changes need to be saved and the orchestrator website needs to be restarted.
    4. The orchestrator website can now be opened and the database will be configured
    5. If the page opens normally the connection to Redis and the connection through the Load balancer is working - it is not needed to configure the second node before opening the page.

Setup secondary node

  1. Install the second node using the command line. The most important command is SECONDARY_NODE=1. Be sure to otherwise give the exact same parameters as used for the installation of the first node.
  2. After the installtion run the sript using the output of the script from the primary node 
    Example:& ".\Configure-PlatformNode.ps1" `
    -mode 'ConfigureSecondary' `
    -websiteName 'UiPath Orchestrator' `
    -nugetPackagesPath '\\fileshare\NugetPackages' `
    -nugetActivitiesPath '\\fileshare\NugetPackages\Activities\' `
    -nugetPackagesApiKey 'EE8B9EF0-DC7F-4AF6-8693-E48EEAC04246' `
    -nugetActivitiesApiKey 'EE8B9EF0-DC7F-4AF6-8693-E48EEAC04246' `
    -encryptionKey 'nHEtV1IbsB/Z5xvpIvkKNkjUWuKsK+2tLWPNgSvgwaE=' `
    -machineKeyDecryptionAlgorythm 'AES' `
    -machineKeyDecryptionKey 'B948A5247C472BA5D71B678ECEC395D861F3449CA85D31E0CF44D9A0ED10956C' `
    -machineKeyValidationAlgorythm 'SHA1' `
    -machineKeyValidationKey 'AC56F985C38FFFC4C0B901049EE78878F63F2B89A3A33C2253A2BDA2918C5323364371A22E8009B392B6839C138DC9E600D77F8F742B28026EF705BF9DF479F6' `
    -redisServer '172.28.4.117' `
    -redisPort 6379
  3. After the script finishes update the web.config file accodrdingly the the primary node.
  4. Connect to the local address of the orchestrator to verfiy that it works, login and create an asset
  5. Login to the local adress of the first node and check if the asset is available.

Orchestrator Configuration

SSL Certificates

  • Certificate must be issued to Loadbalancer DomainName
    • The Orchestrator DN should apear as subject alternative name
  • All Robots are configured to connect to the Loadbalancer
  • The Orchestrator should close port 443 in general and just open it for the Loadbalancer

Web.config

<add key="LoadBalancer.Redis.ConnectionString" value="node1:6379,node2:6379,node3:6379,password=bvbvbbv" />
<add key="LoadBalancer.Redis.ConnectionString" value="node1:6379,node2:6379,node3:6379" />

Load balancer Configuration

Some general foundations:

  • the Robot always initiates the connection to the Orchestrator/Load Balancer to establish a websocket connection (even all this is wrapped with SignalR)
  • Thus the single port 443 needs to be opened for inbound traffic at the Orchestrator only (and Loadbalancer)
  • Hence all the traffic to and from the orchestrator goes through the load balance
  • All Robots access the Loadbalancer - never the Orchestrator directly.

Recommended Loadbalancer by UiPath is F5 big IP:

  • Recommended Balancing Method: Round Robin or Predictive (Preditive performed better in a big HA setup in Japan)

Redis Configuration

Windows Binaries: https://github.com/hteo1337/Redis/tree/master/Redis%20for%20windows

Redis in Windows is well maintained and can be used in windows. However if linux VMs are available is recommended to put the redis nodes on linux.

Needed Ports

  • 6379 (Redis)
  • 26379 (Sentinel)

Redis.Server.conf

  • line 4: bind to real public ip
  • line 12ff: 2x configure slaves

Redis.sentinel.conf

  • line 2: bind to real public ip
  • line 7: change to master node
Each Orchestrator has all Redis-Nodes (Masters and Slaves) configured as valid Redis Nodes, thus no Loadbalancer for Redis is needed

Database Configuration

  • Recommended is to use an already established, high available sql server cluster. If it is not available, it needs to be created or the database gets the single point of failure.

Shared Drive

  • In most companies Windows/CIFS/SMB shares are already available. They are needed as a central place where the Orchestrators store the process packages.
  • Those SMB shares should be high available too as they get the single point of failure if not.

The content by label feature displays related articles automatically, based on labels you choose. To edit options for this feature, select the placeholder below and tap the pencil icon.




    • Related Articles

    • 407 authentication required

      Introduction This article is about a possible fix of a package error. The Screenshot shows the Error message of an installed package. The error code is "407" and occurs because the authorization to UiPath Gallery and thus to all its packages was not ...