SquidNet Network Render Manager
SQUIDNET User’s Guide Network Render Manager ROUGH DRAFT RELEASE *** WORK IN PROGRESS DOCUMENT *** 22 July 2014
Chapter 1 1
SquidNet Network Render Manager
TABLE OF CONTENTS TABLE OF CONTENTS ................................................................................................. 2 CHAPTER 1 1.1 1.1 1.2 1.3
INTRODUCTION ................................................................................ 8
OVERVIEW ......................................................................................................................................................................... 9 TERMINOLOGY .................................................................................................................................................................. 10 TECHNICAL OVERVIEW ....................................................................................................................................................... 11 FEATURES OVERVIEW......................................................................................................................................................... 12 1.3.1 High-level ............................................................................................................................................................... 12 1.3.2 Render Farm Management .................................................................................................................................... 12 1.3.3 Graphical User Interface ........................................................................................................................................ 12 1.3.4 Job Processing ........................................................................................................................................................ 13 1.3.5 Render Node Management.................................................................................................................................... 13 1.3.6 Power Management .............................................................................................................................................. 14 1.3.7 Transactions ........................................................................................................................................................... 14
CHAPTER 2
THE BASICS ...................................................................................... 15
CHAPTER 3
QUICK START TUTORIAL ............................................................. 20
2.1 KEY WORDS...................................................................................................................................................................... 16 2.2 WORKFLOW ..................................................................................................................................................................... 16 2.2.1 Processing Workflow.............................................................................................................................................. 16 2.2.2 User Workflow ....................................................................................................................................................... 18 2.2.3 Configuration Workflow......................................................................................................................................... 19
3.1 BEFORE WE GET STARTED ................................................................................................................................................... 21 3.2 OVERVIEW ....................................................................................................................................................................... 21 3.3 ASSUMPTIONS .................................................................................................................................................................. 22 3.4 INSTALLING 3RD PARTY APPLICATION(S) ................................................................................................................................. 22 3.5 SETTING UP A NETWORK FOLDER.......................................................................................................................................... 22 3.6 SETUP TUTORIAL SCENES..................................................................................................................................................... 23 3.7 SETTING UP SSA ACCOUNT ................................................................................................................................................. 23 3.8 INSTALLING SQUIDNET ....................................................................................................................................................... 23 3.9 CONFIGURE RENDER FARM NODES........................................................................................................................................ 23 3.10 REGISTER LICENSE.............................................................................................................................................................. 24 3.11 SET UP DEFAULT CONFIGURATION PATH ................................................................................................................................ 24 3.12 SET UP JOB TEMPLATES....................................................................................................................................................... 25 3.13 SET UP APPLICATION PROFILES ............................................................................................................................................. 25 3.14 CREATE PROJECT FOLDER .................................................................................................................................................... 27 3.15 SUBMIT JOB REQUESTS ....................................................................................................................................................... 27 3.15.1 After Effects Job ................................................................................................................................................... 27 3.15.2 Maya Job .............................................................................................................................................................. 28 3.15.3 Modo Job.............................................................................................................................................................. 30 3.16 MONITORING RENDER FARM ............................................................................................................................................... 30 3.16.1 GUI monitoring..................................................................................................................................................... 31 3.16.1.1 Project View ................................................................................................................................................ 32 3.16.1.2 Job Queue View .......................................................................................................................................... 32 3.16.1.3 Job Slices ..................................................................................................................................................... 33 3.16.1.4 Application log ............................................................................................................................................ 34 3.16.1.5 Render Farm View ....................................................................................................................................... 34 3.16.1.6 Farm Work Queue View .............................................................................................................................. 35 3.16.1.7 Image Preview View .................................................................................................................................... 35 3.17 VIDEO FROM IMAGE SEQUENCES .......................................................................................................................................... 35 3.17.1 Playback application ............................................................................................................................................ 36 3.17.2 After Effects Video ............................................................................................................................................... 36 3.17.3 Maya Video .......................................................................................................................................................... 37 3.17.4 Modo Video .......................................................................................................................................................... 38 3.18 MODO TILE RENDERING ..................................................................................................................................................... 38 3.19 USING SKIP FRAME............................................................................................................................................................ 41 3.20 USING FRAME-CHECK ........................................................................................................................................................ 41 3.21 USING IMAGE DISPLAY ....................................................................................................................................................... 42
2
SquidNet Network Render Manager
CHAPTER 4
INSTALLATION ................................................................................ 43
CHAPTER 5
USER INTERFACE OVERVIEW ..................................................... 67
4.1 OVERVIEW ....................................................................................................................................................................... 44 4.2 REQUIREMENTS ................................................................................................................................................................ 44 4.2.1 Hardware set up..................................................................................................................................................... 45 4.2.2 NAS/SAMBA Storage Devices................................................................................................................................. 45 4.2.3 Network configuration ........................................................................................................................................... 45 4.2.3.1 Setting up network shares ............................................................................................................................ 45 4.2.3.1.1 WINDOWS............................................................................................................................................. 45 4.2.3.1.2 Mac OS .................................................................................................................................................. 46 4.2.3.1.3 Linux ...................................................................................................................................................... 46 4.3 OPERATING SYSTEM ACCOUNTS .......................................................................................................................................... 46 4.3.1 Windows ................................................................................................................................................................ 46 4.3.2 Mac OS and Linux................................................................................................................................................... 48 4.4 OPERATING SYSTEM PREPARATION....................................................................................................................................... 48 4.4.1 Windows ................................................................................................................................................................ 48 4.4.2 Mac OS ................................................................................................................................................................... 48 4.4.3 Linux ....................................................................................................................................................................... 48 4.5 INSTALLING SQUIDNET ....................................................................................................................................................... 49 4.5.1 Starting and Stopping ............................................................................................................................................ 49 4.6 NODE CONFIGURATION ...................................................................................................................................................... 49 4.6.1 Master host location .............................................................................................................................................. 49 4.6.2 Setup node types .................................................................................................................................................... 50 4.6.2.1 Setting up master .......................................................................................................................................... 51 4.6.3 Example Configuration........................................................................................................................................... 52 4.6.4 Changing Node Types............................................................................................................................................. 54 4.6.5 Adding nodes from external subnets ..................................................................................................................... 54 4.6.6 Master Configuration ............................................................................................................................................. 55 4.6.6.1 Queuing methods .......................................................................................................................................... 56 4.6.7 Client Configuration ............................................................................................................................................... 56 4.6.8 Slave Configuration ................................................................................................................................................ 57 4.7 LICENSE REGISTRATION ...................................................................................................................................................... 57 4.8 SETTING PREFERENCES ....................................................................................................................................................... 57 4.8.1 Configuration Path ................................................................................................................................................. 57 4.8.2 Job Templates ........................................................................................................................................................ 58 4.8.3 Notifications ........................................................................................................................................................... 58 4.9 RENDERING APPLICATION INSTALLATIONS ............................................................................................................................. 59 4.9.1 Application Path Profiles ........................................................................................................................................ 60 4.9.2 Using Application Path Manager ........................................................................................................................... 62 4.9.3 Using Application Path Scanner ............................................................................................................................. 63 4.9.4 Using Application Profiles ...................................................................................................................................... 63 4.10 TRANSLATION PATH PROFILES ............................................................................................................................................. 64 4.10.1 Translation Path Manager ................................................................................................................................... 64 4.10.2 Using Translation Paths ....................................................................................................................................... 65 4.11 UPGRADING ..................................................................................................................................................................... 65 4.11.1 Remote installation using user interface ............................................................................................................. 65
5.1 LICENSE MANAGER............................................................................................................................................................ 69 5.1.1 Registering Licenses ............................................................................................................................................... 70 5.1.2 Hardware Signatures ............................................................................................................................................. 70 5.2 PREFERENCES ................................................................................................................................................................... 71 5.2.1 Configuration Settings ........................................................................................................................................... 71 5.2.2 Job Submission Settings ......................................................................................................................................... 72 5.2.3 Notifications Settings ............................................................................................................................................. 73 5.3 PROJECT MANAGER........................................................................................................................................................... 74 5.4 APPLICATION PATH MANAGER ............................................................................................................................................ 75 5.4.1 Creating an application profile .............................................................................................................................. 76 5.5 APPLICATION PATH SCANNER .............................................................................................................................................. 77 5.6 PATH TRANSLATION MANAGER ........................................................................................................................................... 77 5.7 RENDER POOL MANAGER ................................................................................................................................................... 79 5.8 IMAGE SEQUENCE VIEWER.................................................................................................................................................. 81 5.9 WINDOW VIEWS............................................................................................................................................................... 81 5.9.1 Project folder .......................................................................................................................................................... 81 5.9.2 Job Queue............................................................................................................................................................... 82 5.9.3 Job Slice Queue....................................................................................................................................................... 83 3
SquidNet Network Render Manager
5.9.4 Render Farm ........................................................................................................................................................... 84 5.9.5 Network Work Queue ............................................................................................................................................ 84 5.9.6 Image Preview........................................................................................................................................................ 85 5.9.7 Render Timeline ..................................................................................................................................................... 85 5.9.8 CPU Usage .............................................................................................................................................................. 86 5.9.9 Network Map ......................................................................................................................................................... 86 5.9.10 Job Stack............................................................................................................................................................... 87
CHAPTER 6 6.1 6.2 6.3 6.4 6.5 6.6 6.7
CONFIGURATION ............................................................................ 88
CONFIGURATION PATH ....................................................................................................................................................... 89 APPLICATION TEMPLATES ................................................................................................................................................... 90 LICENSE REGISTRATION ...................................................................................................................................................... 90 JOB NOTIFICATIONS ........................................................................................................................................................... 90 SETUP DEFAULT PROJECT .................................................................................................................................................... 90 EMAIL SETUP .................................................................................................................................................................... 90 SETTING UP MASTER .......................................................................................................................................................... 90
CHAPTER 7
PROJECT INTERFACE..................................................................... 91
CHAPTER 8
JOB SUBMISSION............................................................................. 93
7.1 PROJECT MANAGER........................................................................................................................................................... 92 7.2 PROJECT FOLDER............................................................................................................................................................... 92
8.1 8.2 8.3 8.4
OVERVIEW ....................................................................................................................................................................... 94 JOB SUBMISSION FORM...................................................................................................................................................... 95 OVERVIEW ....................................................................................................................................................................... 95 PROCESSING PARAMETERS.................................................................................................................................................. 97 8.4.1 Template type ........................................................................................................................................................ 97 8.4.2 Project Folder ......................................................................................................................................................... 97 8.4.3 Scene/Shot name ................................................................................................................................................... 97 8.4.4 Job identifier ........................................................................................................................................................... 97 8.4.4.1 Auto Job Id Generation ................................................................................................................................. 97 8.4.4.2 Use Scene Name ............................................................................................................................................ 97 8.4.4.3 Job Identifier ................................................................................................................................................. 97 8.4.4.4 Job Description .............................................................................................................................................. 97 8.4.5 Basic parameters ................................................................................................................................................... 97 8.4.5.1 Max Slices ...................................................................................................................................................... 97 8.4.5.2 Slice Timeout(multiplier) ............................................................................................................................... 97 8.4.5.3 Slice Timeout(mins) ....................................................................................................................................... 97 8.4.5.4 Slice Fails before Stop ................................................................................................................................... 97 8.4.5.5 Check Output Content................................................................................................................................... 97 8.4.5.6 Frame Check .................................................................................................................................................. 97 8.4.5.7 Use all cores .................................................................................................................................................. 98 8.4.5.8 Start Suspended ............................................................................................................................................ 98 8.4.5.9 Randomize Job Slices..................................................................................................................................... 98 8.4.5.10 Skip Existing Frames .................................................................................................................................... 98 8.4.5.11 Create Ouput Folder ................................................................................................................................... 98 8.4.5.12 Estimate Completion Time .......................................................................................................................... 98 8.4.6 Priority .................................................................................................................................................................... 98 8.4.7 Advanced Options .................................................................................................................................................. 98 8.4.7.1 Dependencies ................................................................................................................................................ 99 8.4.7.2 Email/SMS Notifications ................................................................................................................................ 99 8.4.7.3 Image Display .............................................................................................................................................. 100 8.4.7.4 Memory Options ......................................................................................................................................... 101 8.4.7.5 On Condition Profiles .................................................................................................................................. 101 8.4.7.6 Video Generation ........................................................................................................................................ 101 8.4.8 Render Pools......................................................................................................................................................... 102 8.4.9 Include/Exclude .................................................................................................................................................... 103 8.4.10 Tile Rendering .................................................................................................................................................... 103 8.5 APPLICATION PARAMETERS ............................................................................................................................................... 104 8.5.1 Presets .................................................................................................................................................................. 104 8.5.2 Application ........................................................................................................................................................... 104 8.5.3 Frame List ............................................................................................................................................................. 104 8.5.4 Render Frames ..................................................................................................................................................... 104 8.5.5 Fields .................................................................................................................................................................... 104
4
SquidNet Network Render Manager
8.6 SUBMITTING JOBS ........................................................................................................................................................... 105 8.6.1 Scene preparation ................................................................................................................................................ 105 8.6.2 GUI submission..................................................................................................................................................... 105 8.6.2.1 Image Sequence Rendering......................................................................................................................... 105 8.6.2.2 Tile Rendering ............................................................................................................................................. 105 8.6.2.3 Video Generation ........................................................................................................................................ 105 8.6.3 CLI submission ...................................................................................................................................................... 105 8.6.3.1 Image Sequence Rendering......................................................................................................................... 105 8.6.3.2 Tile Render .................................................................................................................................................. 105 8.6.3.3 Video Generation ........................................................................................................................................ 105
CHAPTER 9
TILE RENDERING ......................................................................... 106
9.1 SUBMITTING TILE RENDER JOB ............................................................................................... ERROR! BOOKMARK NOT DEFINED. 9.1.1 GUI Submission ......................................................................................................... Error! Bookmark not defined. 9.1.2 CLI Submission ........................................................................................................... Error! Bookmark not defined. 9.2 USING TILE RENDER VIEWER .................................................................................................. ERROR! BOOKMARK NOT DEFINED.
CHAPTER 10 FARM MANAGEMENT ................................................................. 108
10.1 RENDER NODES ................................................................................................................... ERROR! BOOKMARK NOT DEFINED. 10.2 JOB QUEUE ......................................................................................................................... ERROR! BOOKMARK NOT DEFINED. 10.2.1 Job Slices ................................................................................................................. Error! Bookmark not defined. 10.2.2 Network Work Queue ............................................................................................. Error! Bookmark not defined. 10.3 RENDER TIMELINE ................................................................................................................ ERROR! BOOKMARK NOT DEFINED. 10.4 LOGGING ............................................................................................................................ ERROR! BOOKMARK NOT DEFINED. 10.4.1 Settings.................................................................................................................... Error! Bookmark not defined. 10.4.2 Node Logs................................................................................................................ Error! Bookmark not defined. 10.4.3 Job Logs ................................................................................................................... Error! Bookmark not defined. 10.4.4 Application Logs ...................................................................................................... Error! Bookmark not defined. 10.5 POWER MANAGEMENT ......................................................................................................... ERROR! BOOKMARK NOT DEFINED. 10.6 CPU USAGE ........................................................................................................................ ERROR! BOOKMARK NOT DEFINED.
CHAPTER 11 CROSS-PLATFORM TUTORIAL ................................................ 110 11.1 OVERVIEW ..................................................................................................................................................................... 111 11.2 ASSUMPTIONS ................................................................................................................................................................ 112 11.3 INSTALLING 3RD PARTY APPLICATION(S) ............................................................................................................................... 112 11.4 SETTING UP A NETWORK FOLDER........................................................................................................................................ 113 11.5 SETUP TUTORIAL SCENES................................................................................................................................................... 113 11.6 SETTING UP SSA ACCOUNT ............................................................................................................................................... 113 11.7 INSTALLING SQUIDNET ..................................................................................................................................................... 113 11.8 CONFIGURE RENDER FARM NODES...................................................................................................................................... 114 11.9 REGISTER LICENSE............................................................................................................................................................ 115 11.10 SET UP DEFAULT CONFIGURATION PATH ...................................................................................................................... 115 11.11 SET UP JOB TEMPLATES ............................................................................................................................................ 116 11.12 SET UP APPLICATION PROFILES................................................................................................................................... 116 11.13 TRANSLATION PROFILES ........................................................................................................................................... 117 11.14 CREATE PROJECT FOLDER.......................................................................................................................................... 117 11.15 SUBMIT JOB REQUESTS ............................................................................................................................................ 118 11.15.1 After Effects Job ............................................................................................................................................... 118 11.15.2 Maya Job .......................................................................................................................................................... 119 11.15.3 Modo Job.......................................................................................................................................................... 121 11.16 MONITORING RENDER FARM .................................................................................................................................... 122 11.16.1 GUI monitoring................................................................................................................................................. 122 11.16.1.1 Project View ............................................................................................................................................ 123 11.16.1.2 Job Queue View ...................................................................................................................................... 123 11.16.1.3 Job Slices ................................................................................................................................................. 124 11.16.1.4 Application log ........................................................................................................................................ 125 11.16.1.5 Render Farm View .................................................................................................................................. 125 11.16.1.6 Farm Work Queue View.......................................................................................................................... 126 11.16.1.7 Image Preview View................................................................................................................................ 126 11.17 VIDEO FROM IMAGE SEQUENCES ............................................................................................................................... 127 11.17.1 Playback application ........................................................................................................................................ 127 11.17.2 After Effects Video ........................................................................................................................................... 127 11.17.3 Maya Video ...................................................................................................................................................... 128 11.17.4 Modo Video ...................................................................................................................................................... 129
5
SquidNet Network Render Manager
11.18
MODO TILE RENDERING .......................................................................................................................................... 130
CHAPTER 12 TRANSACTIONS............................................................................ 133 CHAPTER 13 COMMAND LINE INTERFACE.................................................... 134 13.1 COMMONLY USED COMMANDS ......................................................................................................................................... 135 13.2 STANDARD COMMAND LINE OPTIONS.................................................................................................................................. 135 13.3 JOB RELATED COMMAND LINE OPTIONS ............................................................................................................................... 153
CHAPTER 14 HOW-TO ......................................................................................... 179 14.1 INSTALLATION................................................................................................................................................................. 180 14.1.1 How do I upgrade the render farm with a single command? ............................................................................ 180
CHAPTER 15 EXAMPLES ..................................................................................... 181 15.1 JOB SUBMISSION............................................................................................................................................................. 182 15.1.1 3DSMAX ............................................................................................................................................................. 182 15.1.1.1 User Interface............................................................................................................................................ 182 15.1.1.2 Command Line .......................................................................................................................................... 182 15.1.2 After Effects........................................................................................................................................................ 183 15.1.2.1 GUI............................................................................................................................................................. 183 15.1.2.2 Command Line .......................................................................................................................................... 183 15.1.3 Blender3D ........................................................................................................................................................... 183 15.1.3.1 GUI............................................................................................................................................................. 183 15.1.3.2 Command Line .......................................................................................................................................... 183 15.1.4 Cinema 4D .......................................................................................................................................................... 184 15.1.4.1 GUI............................................................................................................................................................. 184 15.1.4.2 Command Line .......................................................................................................................................... 184 15.1.5 FFMPEG Video .................................................................................................................................................... 184 15.1.5.1 GUI............................................................................................................................................................. 184 15.1.5.2 Command Line .......................................................................................................................................... 184 15.1.6 LightWave .......................................................................................................................................................... 185 15.1.6.1 GUI............................................................................................................................................................. 185 15.1.6.2 Command Line .......................................................................................................................................... 185 15.1.7 Luxology Modo................................................................................................................................................... 185 15.1.7.1 GUI............................................................................................................................................................. 185 15.1.7.2 Command Line .......................................................................................................................................... 185 15.1.8 Maya Mental Ray Renderer ............................................................................................................................... 186 15.1.8.1 GUI............................................................................................................................................................. 186 15.1.8.2 Command Line .......................................................................................................................................... 186 15.1.9 Maya Software Renderer ................................................................................................................................... 186 15.1.9.1 GUI............................................................................................................................................................. 186 15.1.9.2 Command Line .......................................................................................................................................... 186 15.1.10 Maya Renderman Renderer ............................................................................................................................. 187 15.1.10.1 GUI .......................................................................................................................................................... 187 15.1.10.2 Command Line ........................................................................................................................................ 187 15.1.11 Maya V-Ray Renderer ...................................................................................................................................... 187 15.1.11.1 GUI .......................................................................................................................................................... 187 15.1.11.2 Command Line ........................................................................................................................................ 187 15.1.12 Nuke ................................................................................................................................................................. 187 15.1.12.1 GUI .......................................................................................................................................................... 187 15.1.12.2 Command Line ........................................................................................................................................ 188 15.1.13 Terragen ........................................................................................................................................................... 189 15.1.13.1 GUI .......................................................................................................................................................... 189 15.1.13.2 Command Line ........................................................................................................................................ 189 15.2 TILE RENDERING ............................................................................................................................................................. 189 15.2.1 3DSMAX ............................................................................................................................................................. 189 15.2.1.1 GUI............................................................................................................................................................. 189 15.2.1.2 Command Line .......................................................................................................................................... 189 15.2.2 Blender3D ........................................................................................................................................................... 189 15.2.2.1 GUI............................................................................................................................................................. 189 15.2.2.2 Command Line .......................................................................................................................................... 189 15.2.3 LightWave .......................................................................................................................................................... 190 15.2.3.1 GUI............................................................................................................................................................. 190
6
SquidNet Network Render Manager
15.2.3.2 Command Line .......................................................................................................................................... 190 15.2.4 Luxology Modo................................................................................................................................................... 190 15.2.4.1 GUI............................................................................................................................................................. 190 15.2.4.2 Command Line .......................................................................................................................................... 190 15.2.5 Maya .................................................................................................................................................................. 190 15.2.5.1 GUI............................................................................................................................................................. 190 15.2.5.2 Command Line .......................................................................................................................................... 190 15.2.6 Terragen ............................................................................................................................................................. 190 15.2.6.1 GUI............................................................................................................................................................. 190 15.2.6.2 Command Line .......................................................................................................................................... 190 15.3 GENERIC JOB .................................................................................................................................................................. 191 15.3.1.1 GUI............................................................................................................................................................. 191 15.3.2 Command Line ................................................................................................................................................... 191 15.4 TRANSACTIONS ............................................................................................................................................................... 191 15.4.1.1 GUI............................................................................................................................................................. 191 15.4.2 Command Line ................................................................................................................................................... 191
CHAPTER 16 FAQS ................................................................................................ 193 16.1 CONFIGURATION QUESTIONS ............................................................................................................................................ 194 16.1.1 How do I relocate the master node? .................................................................................................................. 194 16.2 LICENSING...................................................................................................................................................................... 194 16.2.1 What types of licenses does SquidNet require? ................................................................................................. 194 16.2.2 Do SquidNet license ever expire? ....................................................................................................................... 194 16.2.3 Are SquidNet license good for all future updates? ............................................................................................ 194 16.3 INSTALLATION................................................................................................................................................................. 194 16.4 CONFIGURATION ............................................................................................................................................................. 194 16.4.1 Why is a render node’s state set to "Multiple IP Detection" ? ........................................................................... 194 16.5 JOB SUBMISSION............................................................................................................................................................. 195 16.6 JOB MANAGEMENT ......................................................................................................................................................... 195 16.6.1 Is the 'invalid character string detected'…'string too long'… message in the job log a problem? .................... 195 16.6.2 Why do job slices get requeued when a low memory condition is detected? .................................................. 195 16.7 RENDER FARM MANAGEMENT .......................................................................................................................................... 195 16.7.1 What is the easiest way to remove a local node from rendering part of a job? ............................................... 195
CHAPTER 17 TROUBLESHOOTING .................................................................. 196 17.1 INSTALLATION................................................................................................................................................................. 197 17.2 CONFIGURATION ............................................................................................................................................................. 197 17.3 JOB SUBMISSION............................................................................................................................................................. 197 17.4 JOB MANAGEMENT ......................................................................................................................................................... 197 17.5 FARM MANAGEMENT ...................................................................................................................................................... 197 17.6 NETWORKING ................................................................................................................................................................. 197 17.7 197
CHAPTER 18 APPENDICES.................................................................................. 198
7
Squidnet Network Render Manager
Chapter 1 INTRODUCTION
8
SquidNet Network Render Manager
1. INTRODUCTION
1.1 Overview SquidNet is an advanced render farm management system that controls the distribution and processing of single-frame, multiframe, video generation and workflow management requests. A SquidNet render farm consists of submitting workstations (clients), a workflow manager (master) and rendering/processing engines (slaves). Jobs are submitted from clients using SquidNet’s graphical user (GUI) and command line (CLI) interfaces. Once received, the master persistently stores these jobs in its local render queue and intelligently breaks up the requests into smaller segments that get evenly distributed to available render farm processing (rendering) nodes. These small segments, called job slices, contain rendering instructions for the processing nodes. While render nodes are processing requests, the master idly stands by waiting for job completion responses from these nodes or for incoming job requests from submitting client nodes. The Master can be run on any networked computer (Windows, Mac, Linux), uses very little system resources and is heavily multi-threaded to support thousands of queued jobs and hundreds of client workstations and render nodes. Client nodes can also run on any computer platform and can co-exist with your 3D applications. Slave nodes are the render farm work horses and can reside on any platform as well. Although SquidNet’s primary role is to manage render farm processing operations, it’s also a powerful tool for workflow management. With SquidNet’s transaction framework, you can schedule batch jobs that contain user-defined scripts than can be executed on any render node at any time of the day. Transactions also allow you to queue job request chains, video generation, output folder archiving and image transfer operations to remote locations using secure communications channels (SFTP/FTP, SSH, OpenSSL, etc…).
9
SquidNet Network Render Manager
Figure 1 Typical SquidNet render farm configuration
1.1 Terminology Term Application Path Application Profile CLI Client Daemon
Frame Frame Check GUI Job Master
MDI NAS Node
Pickup Frame
Pool Render Farm
Description The installation folder for a 3D or compositing application. This path typically contains the absolute path to the rendering application’s rendering program. Contains a list of nodes and the application paths for rendering applications. Command Line Interface. For SquidNet, this is the “sqn” command line interface. A workstation computer used to submit job requests to the render farm. Clients can be configured to optionally render jobs. Background service on Linux or mac OS platforms. Each SquidNet server runs as a daemon on these operating systems. On WINDOWS platforms, daemons are called services. A numbered image within an animation clip. The output of a rendered frame is an image file. A feature that automatically creates single frame jobs for images that are missing or incorrectly rendered. Graphical User Interface. The main SquidNet interface used on clients and the master node. Slave nodes don’t need access to the GUI. A processing request within a render farm. Typically, it’s a request to render a single frame or image sequence. The render farm computer that manages all render farm operations. Can be configured to optionally render jobs (although, depending on workflow, this is not recommended) Multiple Document Interface. SquidNet’s main GUI is based on an MDI application format. Network Attached Storage. NAS drives are recommended for storage of all rendering content and image output. A computer configured to run within a render farm configuration. Can be a client, master or slave computer. Synonymous with an individual computer system. A node can be a Windows, Mac or Linux computer system. Jobs created by SquidNet (as apposed to user requests) and added to the render farm queue. Typically, these types of jobs are created when the Frame-Check feature is enabled. Groupings of render nodes that are associated by a common name and perform specific processing operations. A multi-computer network that’s dedicated to the processing (rendering) of
10
SquidNet Network Render Manager
Render node Rendering Application SAN Skip Frame Slave Slice sqn SSA Task Transaction Translation Profile View WOL
computer-generated imagery (CGI) for the film, television, gaming, architectural and multi-media industries. A computer that processes user requests. Typically, this is a slave node but client and master nodes can be configured to process jobs as well. A 3rd party program used to process (render) scene content. Example rendering applications include Maya, 3DSMAX, mod, LightWave and Terragen. Storage Area Network. Typically used in high-performance environments. A feature that prevents the re-processing of frames for images that have already been rendered. A computer selected to only perform rendering operations. The portion of a job that gets distributed to render nodes. Slices contain processing instructions for render nodes. SquidNet command line interface program. Avalailable on all supported platforms. SquidNet Service Account. The administration account that the SquidNet service runs under (Windows only). A transaction component that can includes frame renders, script execution, video creation, etc… A sequential list of tasks performed by the SquidNet master. Contains the common network share mappings for Windows, Mac OS and Linux platforms. An open window within the client area of the main user interface. Wake-on-Lan network protocol that sends messages to computers to bring them out of a sleeping or hibernating state.
Table 1 SquidNet terminology
1.2 Technical Overview A SquidNet render farm consists of 3 major subsystems:
Master (render farm controller): o Oversees all render farm operations (render jobs, transactions, etc…). o Manages render farm job queue. o Monitors render node activity. o Distributes job instructions (slices) to rendering nodes. o Can optionally be configured to process jobs (although not recommended for heavy workloads). Client (submission workstation): o Submits job requests to render farm queue. o Render farm can contain hundreds of client workstations. o Can optionally be configured to render jobs: Never (submit mode only) When users are logged out. When users are logged in but execute at a low processing priority so as to not disturb desktop performance. When screen saver is active (Mac OS only). Slave (rendering engine): o Sole purpose is to process rendering requests. o Render farm can contain hundreds of slave nodes.
From a high level, a SquidNet render farm can exist in any environment where computers are networked together. With SquidNet, the following render farm configurations are possible:
WINDOWS only: Environment where all computers use Microsoft Operating Systems (XP, Vista, WIN7, WIN8, WINDOWS Server, etc…) Mac OS only: Environment where all computers use Apple’s Mac OS Operating System (Snow Leopard, Lion, Mountain, Lion, Mavericks, etc…). Linux only: Environment where all computers use some variant of Linux (RedHat, Fedora, Ubuntu, CentOS, OpenSUSE, etc…) Cross-platform: Environment where different operating systems co-exist on the same physical network.
Each SquidNet node can be configured to run on any operating system. For example, in a cross-platform system, Mac OS client workstations can submit jobs to the SquidNet master running on a Linux distribution and it can send jobs to Windows-based render nodes. With SquidNet any render farm configuration is possible.
11
SquidNet Network Render Manager
From a networking perspective, all SquidNet nodes on the same subnet automatically find each other so there’s never a need to mess with complicated configuration files. For nodes on external subnets, SquidNet provides GUI/CLI options for connecting those nodes to the SquidNet master .
1.3 Features Overview 1.3.1 High-level
Cross-platform support for Windows, Mac OS and most major Linux distributions. Intuitive MDI (multiple-document interface) Graphical User Interface (GUI). Command Line Interface (CLI). Software Development Kit (SDK). Simple initial installation with single-command software updates for any size render farm. Easy configuration thru GUI or CLI interfaces. No configuration files to mess with ever. If needed, easy re-configuration thru GUI or CLI. Single and multi-frame rendering. Support for thousands of queued jobs. Support for hundreds of client workstations and slave render nodes. Multi-threaded master controller for optimal workflow management efficiency. Flexible architecture to support future expansion. Secure storage of all sensitive information using AES-256 bit encryption. Scalable architecture to support any size render farm. Reliable subsystems that allow render farm to run flawlessly without issue.
1.3.2 Render Farm Management
Standard render farm management operations (submit, suspend, resume, cancel, etc...) Real-time job queue monitoring (state, priority, percentage complete, completion status, etc…) Real-time render node management and monitoring (idle, processing, locked, etc…) GUI and CLI management operations.
1.3.3 Graphical User Interface
Ease of use: o Consistency: The same interface runs on Windows, Mac OS and Linux platforms. o Multiple Document Interface: Persistent user-defined window layouts. o Tracking: History queues for jobs and job slices. Render farm monitoring: o Project view: Lists all jobs that belong to a specific project folder. o Job Queue view: Lists all render queue jobs. o Render Farm view: Lists all render farm nodes and their operating status. o Network Work Queue view: Lists all active render nodes and the jobs that they’re working on. o Transaction Queue view: Lists all active transactions. o Render Timeline: Shows render farm processing status for the last 24 hours. o CPU usage: Shows CPU usage for entire render farm or for individual nodes. o Image Preview: Preview rendered images by clicking on job in project folder, job queue, job slice and network work queue views. o Network Map: Shows network hierarchy of all render farm nodes. o Job Queue Stack: Shows job queue entries listed by priority. o Job Queue Matrix: Shows dependencies for each queue job. Management: o Preferences: Set up user preferences. o License Manager: Manages all registered licenses. o Project Manager: Project folder management. o Licenses in use: Displays status of allocated licenses. o Render Pool Manager: Gtoups nodes together to process specific jobs. o Image and Sequence Viewer: Display images and playback image sequence animations. o Application Path Manager: Declare paths for render farm applications. o Application Path Manager Scanner: Scan render farm looking for installed applications. o Translation Path Manager: Declare mappings between Windows, Mac OS and Linux paths. o Mapped Drive Manager: System-wide auto-mapping of Windows mapped drives. o Environment Variable Manager: Define environment variables that are to be defined at render time.
12
SquidNet Network Render Manager
Window Management: o Cascaded window views: Arrange windows in a parent-child hierarchy. o Tiled window views: Arrange windows side-by-side. o Tabbed window view: Arrange each window so that each takes up the entire client area. o Full screen mode: Maximize GUI to extents of host monitor. o Persistent window configurations: Save and restore user-defined window layouts.
1.3.4 Job Processing
Request Types: o Multi-frame rendering: Render multiple frames on multiple computers. o Single frame (tile) rendering: Render a single frame on multiple computes o H.264/MPEG4 Video generation: Generate video files from rendered image sequences. Submission options: o Job id assignment: Create jobs with names (ids) up to 512 characters in length. o Shot name assignment: Group and display jobs by shot name or job id. o Color Id: Easily identify jobs within window views by assigning a specific color. o Future times: Start a newly submitted job at a later date. o Priority assignment: 25 different processing priorities (0-24). “0” being the highest priority and 24 being the lowest. o Job dependencies/chaining: Parent-child job dependencies to any depth. o Render Pools: Assign jobs to render on specific groups of render nodes. o Include/Exclude: Assign jobs to ONLY render on specific nodes or to NOT render on specific nodes. o Job Description: Provide brief description of job request. o Max slices: Set the maximum number of concurrently active job slices. o Job slice timeout multiplier: Automatically requeue any job slice that exceeds the average processing time of the other slices within the same job. o Job slice timeout in minutes: Automatically requeue any job slice that exceeds a specific processing time. o Job slice error max: Set the maximum number of allowed per-job slice errors before job processing is halted. o Use all available cores: Application specific option to use all available processing cores when rendering slice. o Slice load balancing: Auto-calculation of frames-per-slice setting to maximize render node performance. o Randomize job slices: Randomly process job slices (as apposed to processing sequentially (default)). o Frame check: Automatically requeue missing or corrupted output frames. o Skip frame: Don’t process frames that already exist in output folder. o On Condition Profiles: Perform specific actions (requeue, stop, etc…) on the outcome of completed job slices. o Email and SMS notifications: Email/SMS notifications for specific job processing events. o H.264/MPEG4: Auto-generate video files from rendered image sequences. o Image Display: Spot check rendered images. o Rendering account: WINDOWS-only option to render job slices using a specific administration account. Management operations: o Real-time priorities: Change priorities of active job queue entries. o Suspend/Resume: Temporarily pause and restart job queue entries. o Cancel: Delete/remove job queue entries. o Restart: Re-queue completed or errored out job queue entries. o Real-time log: Monitor job activity from text log. o Tile Render Viewer: View image progress of tile render jobs. o Memory usage: Automatically requeue slices that exceed specified memory limits. o Render Report: Text file based report of all render jobs. Optimized job distribution: o Large queue support: Support for thousands of queued jobs. o Smart wakeup: Automatically “wake up” render farm nodes when job is submitted. o Efficiency: Minimal delay between slice distributions: Maximizes efficiency of render nodes. o Smart job distribution: Automatic distribution prioritization to most powerful render nodes.
1.3.5 Render Node Management
Render Pools: Group render nodes to process specific types of jobs. Node Type: Any node can be configured to act as a Master, client or slave. No software re-installation required. Alias names: Assign a unique name to each node. Multi-subnet support: Add or remove nodes from external subnets. Pool Assignment: Add or remove nodes from specific render pools.
13
SquidNet Network Render Manager
Software update: Update SquidNet software versions remotely one at a time or all at once. Logging: Real-time activity logging. Load balancing: Manually redistribute selected job slices to other nodes. Power Management: Power-up/restart and shutdown entire render farm or selected nodes. Wake-On-LAN (WOL) support: Remotely power on render nodes. Activity schedule: Schedule render farm nodes to process jobs at certain times of the day. Lock/unlock: Remove (or re-include) nodes to render farm pipeline. Exceptions log: Render farm exception and error logging. Job history: Show job and work queue activity on each render node.
1.3.6 Power Management
Auto-shutdown: Automatically shut down render nodes that have been idle for a specified period of time. Manual shutdown of entire render farm or selected nodes. WOL support from GUI or CLI. Automatic WOL messages on job submission.
1.3.7 Transactions
Combine multiple render jobs into a single request. User-defined script execution. Output folder archiving. H.264/MPEG4 Video creation from output image sequences. Output folder archiving with AES 256-bit password protection.
14
SquidNet Network Render Manager
Chapter 2 The Basics
15
SquidNet Network Render Manager
2. The BASICS Before we get started with learning the ins and outs of SquidNet, let’s take a moment to learn some of the SquidNet’s basic concepts.
2.1 Key words SquidNet’s key words include:
GUI: SquidNet’s Graphical User Interface. May also be refered to as the “UI”. The GUI has the following subwindows (amoung others): o Job Queue: Lists all active jobs. o Render Farm: Lists all registered render nodes. o Network Work Queue: Lists active (processing) render nodes and shows the jobs that are being processed. o Project: Lists all jobs assigned to the current project. CLI: SquidNet’s Command line Interface. Refered to as “sqn” in this document. Configuraration Folder: Common storage for all GUI settings, project folders, job profiles, etc.. To avoid duplicating GUI settings, set configuration path to a network folder that all submitting nodes have access to. Application Profile: Node or operating system specific application installation information. Contains absolute path of rendering application’s processing program. For example, After Effects uses aerender.exe to process jobs, Maya uses render.exe and 3DSMAX uses 3dsmaxcmd.exe Translation Profile: Contains common network folder mappings as seen by different operating systems (Windows, Mac OS and Linux). Basically, the method used by SquidNet to access the same network share folder between different operating systems. Render node: Applies to any computer node on the render farm including clients, slaves and the master node. Client: Computer used to submit render requests (jobs) to the render farm, Master: The render farm’s management node. Slave: Computers used to process render requests (jobs). Job: User processing request submitted to the farm. Can originate from GUI or from CLI. Contains application processing information (scene file, frames to render, output folder, etc…) and instructions (max jobslices, pools, etc…) Job Profile: Saved copy of submitted job that can be requeued. Listed in Project view in GUI. Job Template: Application and job type. Example: Maya, After Effects, Maya, etc… Slice: Rendering instructions for render nodes. Includes start frame, end frame, output folder, etc…) Derivede from user’s original job request. Activity Logging: o Render Node Log: Render node’s activity log. Unique to each render node. o Job Log: Job activity log. Unique to each queued job. o Slice (Application) Log: Job Slice (application) activity log. Unique to each job slice. Contains the rendering application’s text output. Job Queue: Managed storage of queued jobs. Render Pool: Assignment and grouping of specific render nodes for the sole purpose of processing specific jobs. By default, all render nodes belong to the “NETWORK” pool. Transaction: Special request that contains multiple tasks that are dependent on each other. Output Folder: Network folder where rendered content will be written to.
2.2 Workflow 2.2.1 Processing Workflow
16
SquidNet Network Render Manager
Figure 2 SquidNet Processing Workflow The figure above provides a high-level description of SquidNet’s processing pipeline: 1. 2. 3.
4. 5. 6.
Client UI configuration information is stored in configuration folder that’s accessible by all job submitting nodes (ie; networked). Clients (GUI or CLI) submit jobs to the render farm. GUI configuration information and settings are read from configuration folder. All jobs are submitted to Master node which: a. Receives jobs requests in it’s input queue. b. Validates job quest (job type, rendering parameters, etc…) . c. Adds job to main processing queue (job queue). d. Evenly divides job request into slices (as instructed by job request). e. Checks availability and status of render nodes. f. Distributes job slices to render nodes. g. Waits for new requests from clients or for completion messages from rendering nodes. Clients monitor job queue activity from GUI/CLI interfaces. Upon job completion, Master node removes job entries from job queue. Images/frames are rendered to output folder.
17
SquidNet Network Render Manager
2.2.2 User Workflow
Figure 3 SquidNet User Workflow The figure above shows the job workflow from the user’s perspective: 1. 2. 3. 4. 5. 6.
Named projects are stored in configuration folder (preferably on a network share). Within the named projects, job profiles are stored. Job profiles contain application type (template), scene path, output folder and other processing parameters required for render farm processing. Users open job profiles stored in project folders and submit them to the render farm (ie; master node). The Master divides up the job request into smaller more-managable fragments called slices. Each slice contains processing instructions for the rendering nodes. The Master distributes slices to available render nodes. Each node renders scene frames (images) to specified network storage device.
18
SquidNet Network Render Manager
2.2.3 Configuration Workflow
Figure 4 SquidNet Configuration Workflow The figure above shows SquidNet workflow configuration. 1.
2. 3.
The configuration folder contains the following: a. Application Profiles: Local folder installation paths of rendering applications. b. Translation Profiles: Used in mixed operating system environments. Contains network share mappings between differerent operating systems. c. Project folders: Unique folders that contain job profiles. d. All other settings that can be shared between submitting clients. e. Note: Backend render nodes (master and slaves) never have access to configuration folder items. Submitting client nodes reference settings stored in configuration folder. Submitting client nodes embed processing instructions in job requests.
19
SquidNet Network Render Manager
Chapter 3 QUICK START TUTORIAL
20
SquidNet Network Render Manager
3. QUICK START TUTORIAL
3.1 Before we get started This section will provide a tutorial on how to set up SquidNet on a simple all-Windows render farm. If you’re interested in setting up a cross-platform (Windows, Mac OS, and Linux) render farm, refer to the “Cross-Platform Tutorial” chapter in this document.
3.2 Overview
Figure 5 Quick Start Render Farm Network This tutorial will demonstrate how to get SquidNet up and going as quickly as possible by guiding you thru some basic installation, configuration, job submission, and render farm management operations. The tutorial will use After Effects, Maya and Modo rendering applications on an all Windows render farm. To give you a quick understanding on how to set up SquidNet, the following tasks will be performed:
21
SquidNet Network Render Manager
Prepare After Effects, Maya and Modo for network rendering. Setup network shares for scene content and rendered image storage. Copy sample AE, Maya and Modo scene files from the SquidNet installation folder onto network share folders. Create SquidNet SSA account on all Windows nodes (clients, master and slaves). Install SquidNet on all nodes. Configure master, client and slave nodes from GUI and CLI. Setup default configuration path. Setup application profiles that tell SquidNet where rendering applications are located. Submit rendering jobs from GUI and CLI interfaces. Monitor job progress using tools from GUI and CLI interfaces. Create H.264/MPEG4 video of animation sequences from GUI and CLI interfaces. Submit tile rendering job. Using skip-frame feature. Using frame-check feature. Using image-display feature.
3.3 Assumptions For this tutorial, the following assumptions are made:
Project folders will be stored on a network share: \\raid-server00\volume_1\SquidNet Jobs will be submitted from a SquidNet client node called SQUIDNET-CLIENT. The master node will be named SQUIDNET-MASTER. The master will not be processing jobs. There will be 5 slave nodes. The slaves will be named SQUIDNET-SLAVE[01-05]. Rendered images and videos will be saved to \\raid-server00\volume_1\SquidNet\Tutorial-Images folder.
3.4 Installing 3rd Party Application(s) rd
SquidNet requires that all 3 party rendering applications (Maya, After Effects, LightWave, modo, etc…) be installed on every render node. This is necessary because the local SquidNet service must have access to the application’s rendering program. The rendering program is the command line tool that SquidNet calls to process scene renders. Most 3rd party applications provide a separate renderer program that resides in the same folder as the main application GUI. In most cases, the command line renderer program is different than the main application GUI. The table below shows the renderer programs supplied by various 3D and compositing applications: 3D/Compositing Application After Effects Autodesk 3DSMAX Autodesk Maya Blender3D CINEMA 4D LightWave Luxology Modo Nuke Terragen V-Ray Standalone XSI/SoftImage
WINDOWS RENDERER APP aerender.exe 3dsmaxcmd.exe render.exe blender.exe "CINEMA 4D.exe" lwsn.exe modo_cl.exe nuke.exe tgdcli.exe vray.exe xsibatch.bat
Mac OS RENDERER APP aerender render blender “CINEMA 4D” lwsn modo_cl "Terragen 2" vray -
Linux RENDERER APP Render blender modo_cl vray -
Table 2 3rd Party Application Rendering Programs (not all applications are listed) The highlighted renderers above are what SquidNet will use to render job requests in this tutorial. When you get to the application path manager step, you'll need to tell SquidNet where these applications are installed on your render farm by specifying the paths in your application profile definitions. (More on that in the following sections)
3.5 Setting up a network folder Make sure all AE, Maya and Modo scene files can be accessible from the network share \\raid-server00\volume_1\SquidNet by opening a DOS shell and issuing the following command: > dir \\raid-server00\volume_1\SquidNet Consult with your IT professional if you’re having problems setting up the network share.
22
SquidNet Network Render Manager
3.6 Setup tutorial scenes In SquidNet's installation folder, copy the "Tutorials" folder (including sub-folders) to the network share folder. When the copy is complete, you should have the following folder:
\\raid-server00\volume_1\SquidNet\Tutorials
3.7 Setting up SSA account On Windows render nodes, create an administrator account (SSA) named “SquidNet” with a password of your choice. Once the account is created, log out of your current Windows session and log in again using the new SSA account. Make sure you can access any network folders on the network share folders. Try creating and removing some files on the network share while logged in under the SSA account. If you can access the network share without issue, you should be good to go ! If you already have an administrator account that can act as the SSA, you’re free to use that account. Just make sure that the account can access your network share and that you use the same account name and password on all Windows farm nodes.
3.8 Installing SquidNet Using the instructions that are available on the SquidNet website, install SquidNet on each render farm node:
SquidNet Installation Guide for Windows. (open) SquidNet Installation Guide for Linux. (open) SquidNet Installation Guide for Mac OS. (open)
On Windows nodes, make sure you enter the newly created SSA account when prompted.
3.9 Configure render farm nodes
Figure 2 Initial Render Farm Configuration At this point you should have SquidNet installed on all you render nodes. On the master node, start the main SquidNet GUI and open up the Render Farm Nodes view from the main menu tool bar (Network->Render Farm ...). IMPORTANT: Node configuration must be done on the master computer. Make sure you have the columns displayed as shown above. If not, enable any missing columns by right-clicking on the Render Farm view and selecting the "Select Columns..." option from the popup menu. The next steps will configure each render node. Do the following in order: 1. 2. 3.
4. 5. 6.
Log into the SquidNet GUI on the MASTER node. Right-click on SQUIDNET-MASTER node and select Configuration->Set as Master... menu option. Right-click on SQUIDNET-CLIENT node and select Configuration->Set as Client... menu option. a. Since we don't want our client node to render jobs, right-click on SQUIDNET-CLIENT node and select Configuration->Configure Client... menu option. Choose "Never" from dropdown menu. Select all 5 slave nodes, right-click and select Configuration->Set as Slave... menu option. Wait a few seconds for node list to update. At this point all your render nodes should be configured.
23
SquidNet Network Render Manager
If you make a mistake and mis-configuring a node, simply convert the node back to PEER and restart the process. If you want to completely re-configure all nodes, convert all client and slave nodes to PEERS first then convert the master to a PEER. Once all nodes are PEERS, you can then commence to re-configure the new master, client and slave nodes. The GUI commands above could have been done using the following CLI commands: Equivalent CLI Command sqn --resetFarm sqn --masterSet SQUIDNET-MASTER sqn --nodeTypeSet client --node SQUIDNET-CLIENT sqn --clientModeSet never --node SQUIDNETCLIENT sqn --nodeTypeSet slave --node SQUIDNETSLAVE01 sqn --nodeTypeSet slave --node SQUIDNETSLAVE02 sqn --nodeTypeSet slave --node SQUIDNETSLAVE03 sqn --nodeTypeSet slave --node SQUIDNETSLAVE04 sqn --nodeTypeSet slave --node SQUIDNETSLAVE05
Description Resets ALL render farm nodes to PEERs Sets node SQUIDNET-MASTER to be the SquidNet master Sets node SQUIDNET-CLIENT to be a SquidNet client Prevents node SQUIDNET-CLIENT from processing jobs Sets node SQUIDNET-SLAVE01 to be a SquidNet slave Sets node SQUIDNET-SLAVE02 to be a SquidNet slave Sets node SQUIDNET-SLAVE03 to be a SquidNet slave Sets node SQUIDNET-SLAVE04 to be a SquidNet slave Sets node SQUIDNET-SLAVE05 to be a SquidNet slave
Figure 3 Configured Render Nodes If at this point the nodes don’t look like they do in the figure above, please go back to see if you missed a step.
3.10 Register license If you have a SquidNet license, you can register it as follows:
Open GUI on master node. Open License Manager dialog (Help->License Manager...) Press "Register New License" button and cut/paste in your SquidNet license(s). If you have an evaluation license, you’ll need to remove it (right-click “Remove…”) before registering new one.
3.11 Set up default configuration path
Figure 4 Configuration Path
24
SquidNet Network Render Manager
The configuration path is where the SquidNet UI will save all your GUI settings (jobs, presets, pools, etc...). To avoid having to duplicate GUI settings on each submission node, make sure you set the configuration path to a network shared folder. On the master and on every client node, open the Preferences dialog and set the Configuration Path to:
\\RAID-SERVER00\volume_1\Squidnet\SquidNetConfig.
Press the "Apply" button to set the path. * Create the “SquidNetConfig” folder if it doesn’t already exist.
3.12 Set up job templates
Figure 5 Job Templates With the SquidNet preferences dialog still open, select the "Job Submission" option on the left side of the dialog and add the After Effects, FFMPEG, Maya and Modo templates to the right side list. This setting determines which job templates are available in your project window and which applications you’ll be able to submit jobs to.
3.13 Set up application profiles
Figure 6 Application Path Manager We'll need to setup profiles to tell SquidNet where to find our applications so open up the Application Path Manager and register our AE, Maya and Modo render applications. On the client node (SQUIDNET-CLIENT) or master (SQUIDNET-MASTER), start the SquidNet GUI and do the following (remember, because the configuration path on the client and master nodes are set to the same network folder, each GUI will “see” the same settings.:
Open the Application Path Manager dialog (Admin->Path Configuration->Application Path Manager ...)
25
SquidNet Network Render Manager
Create 3 Application profiles with the following settings (see pics below): o Profile Name SQUIDNET-TUT-AFX SQUIDNET-TUT-MAYA SQUIDNET-TUT-MODO
Application aerender.exe render.exe modo_cl.exe
Render Node
Renderer Path C:\Program Files\Adobe\Adobe After Effects CS6\Support Files C:\Program Files\Autodesk\Maya2014\bin C:\Program Files\Luxology\modo\701_sp1
o
We’re purposely using the option but here’s a list of other possible options for the “Render Node” field: : Use the specified application path on all render nodes. Should only be used when all nodes are the same platform AND all nodes have the same application installation path. : The application path only applies to Windows render nodes. : The application path only applies to Mac OS render nodes. : The application path only applies to Linux render nodes. : The application path is only valid for the specified node name. The paths above are assumed so if your installation paths are different, update the application path accordingly. It's also assumed that each slave node has each application installed on its local hard drive. (Note the "" selection for the "Render Node:" field). Note: SquidNet will not submit slices to nodes that are missing render applications. Make sure to "validate" each profile. Press "OK" button to save profile(s)
Figure 7 After Effects Application Profile
Figure 8 Maya Application Profile
26
SquidNet Network Render Manager
Figure 9 Modo Application Profile
3.14 Create project folder All SquidNet render jobs are stored in project folders. So before we can submit our render jobs, we'll need to create a project folder:
Open the Project Manager dialog (Project->Project Manager...) Press the "New Project" button and create a project named "SQUIDNET-TUTORIAL". Note that the new project folder is created in the configuration path that was previously set. Double-click on the project "SQUIDNET-TUTORIAL" in the Project Manager dialog to open a project view for project "SQUIDNET-TUTORIAL"
Figure 10 New Project "SQUIDNET-TUTORIAL"
3.15 Submit job requests
Figure 11 Project "SQUIDNET-TUTORIAL"
3.15.1 After Effects Job We're now ready to formally submit jobs to the render farm. With the view for project "SQUIDNET-TUTORIAL" open in the GUI client area, press the "New Job" button and select the After Effects template from the top-left of the job submission form.
27
SquidNet Network Render Manager
Figure 12 After Effects Job Submission On the job submission form, enter or change the following fields: 1.
2.
3. 4.
On left-side of the form: a. Set the "Job ID" to AFX-JOB#1. b. Select the "Create output folder" checkbox. On right-side of the form, set the following fields: a. Application: $APP(SQUIDNET-TUT-AFX) b. Project File: //RAID-SERVER00/Volume_1/SquidNet/Tutorials/AfterEffects/fire.aep c. Start Frame: 1 d. Ending Frame: 400 e. Frames Per Slice: 1 f. Output Folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/AE g. Output Filename: fire[####].jpg h. Under "Other settings" panel: i. Render Queue Index: 1 Press the "Submit" button. When asked to save job, say "yes"
The AE template settings coincide with the scene settings defined in the After Effects GUI. Equivalent CLI Command sqn --submit --template AE --appPath $APP(SQUIDNET-TUT-AFX) --jobId AFX-JOB#2 --startFrame 1 --endFrame 400 --priority 12 --framePerSlice 1 --createOutputDir --sceneFile //RAID-SERVER00/Volume_1/SquidNet/Tutorials/AfterEffects/fire.aep -outputDir //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/AE --videoFilename "fire[####].jpg" --renderIndex 1 -useAllCores
3.15.2 Maya Job Press the "New Job" button and select the Maya template from the top-left of the job submission form.
28
SquidNet Network Render Manager
Figure 13 Maya Job Submission On the job submission form, enter or change the following fields: 1.
2.
3. 4.
On left-side of the form: a. Set "Job ID" to MAYA-JOB#1 b. Select the "Use All Available Cores" checkbox. c. Select the "Create output folder" checkbox. On right-side of the form, set the following fields: a. Application: $APP(SQUIDNET-TUT-MAYA) b. Project Dir: //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Maya c. Scene: //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Maya/fluidParticleFalls.ma d. Working Directory: //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Maya e. Start Frame: 1 f. Ending Frame: 450 g. Frame Per Slice: 20 h. Under "General Purpose" panel: i. Renderer to Use: Software ii. Output Image Path: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Maya iii. Format: name#.ext iv. Type: png Press the "Submit" button. When asked to save job, say "yes"
Equivalent CLI Command sqn --submit --template maya --jobId MAYA-JOB#2 --useRenderer Software --appPath $APP(SQUIDNET-TUT-MAYA) --sceneFile //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Maya/fluidParticleFalls.ma --projectFolder //RAIDSERVER00/Volume_1/SquidNet/Tutorials/Maya --workDir //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Maya --startFrame 1 --endFrame 450 --framePerSlice 20 --imageFilenameFormat name#.ext --priority 12 --outputDir //RAIDSERVER00/Volume_1/SquidNet/Tutorial-Images/Maya --imagePrefix fluidparticles_ --imageFormat png --useAllCores -createOutputDir
29
SquidNet Network Render Manager
3.15.3 Modo Job Press the "New Job" button and select the Modo template from the top-left of the job submission form.
Figure 14 Modo Job Submission On the job submission form, enter or change the following fields: 1.
2.
3. 4.
On left-side of the form: a. Set "Job ID" to MODO-JOB#1 b. Select the "Create output folder" checkbox. On right-side of the form, set the following fields: a. Application: $APP(SQUIDNET-TUT-MODO) b. Scene File: //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Modo/racecar.lxo c. Important: Press "Scan Modo scene" button to auto-populate some of the fields. d. Start Frame: 1 e. Ending Frame: 300 f. Frame Per Slice: 10 g. Output Image Path: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo h. Image Prefix: racecar_ i. Output Image Format: png j. Render Camera: OVERHEAD Press the "Submit" button. When asked to save job, say "yes"
Equivalent CLI Command sqn --submit --jobId MODO-JOB#2 --template modo --appPath $APP(SQUIDNET-TUT-MODO) --sceneFile //RAIDSERVER00/Volume_1/SquidNet/Tutorials/Modo/racecar.lxo --startFrame 1 --endFrame 300 --framePerSlice 10 --priority 12 -outputDir //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo --imagePrefix racecar_ --imageFormat png -useAllCores --createOutputDir --camera OVERHEAD
3.16 Monitoring render farm At this point, you should have 3 jobs submitted to the render farm job queue. In this section, we'll show you some basics on how to monitor and manage these jobs from both the GUI and CLI interfaces. 30
SquidNet Network Render Manager
On the client or master nodes, start the SquidNet GUI and open the following views:
Image Viewer: Tools->Image Preview.. Project view for project "SQUIDNET-TUTORIAL": Project->Project Manager ... Render Farm Nodes: Network->Render Farm... Job Queue: Tools->Job Queue... Network Work Queue: Network->Network Work Queue...
Arrange and resize views in the GUI client area so that all windows are visible. Alternatively, you can select Window->Tile Windows... to auto-arrange windows for you.
3.16.1 GUI monitoring
Figure 15 SquidNet User Interface There's over a dozen window views that you can select from but for the sake of this tutorial we're going to focus on these:
Project View: Image Preview: Job Queue: Render Farm Nodes: Network Work Queue:
31
SquidNet Network Render Manager
3.16.1.1 Project View
The Project view will contain the jobs that we've already submitted. Once saved, you can always open up any jobs in the project, edit the processing parameters and resubmit back to the farm.
3.16.1.2 Job Queue View
Equivalent CLI Command sqn --jobQueueShow Job Id Position State Progress Frames Slices Avg Slice Time GHZ-Hours Est.Proc.Time Est.Comp.Time --------------------------------------------------------------------------------------------------------------------------------------------------AFX-JOB#2 1 Processing 42% 1-400 400 25s 26.428774 01h.54m.10s 01h.54m.10s MAYA-JOB#2 2 Processing 21% 1-450 23 04m.50s 36.344572 02h.40m.33s 04h.34m.43s
The Job Queue view lists all the jobs in the render farm queue. Press the arrow button in the upper right corner of the view to open the job slice side panel. Double-click any job in the job queue to list the associated job slices.
32
SquidNet Network Render Manager
3.16.1.3 Job Slices
Equivalent CLI Command sqn --jobId MODO-JOB#1 --jobSliceShow # Frames State RENDERNODE TIME GHZ-hrs PRI Memory Used -------------------------------------------------------------------------------------------------1 0-9 Completed RENDERNODE14 01m 28s 0.665084 12 446980 KB 2 10-19 Completed RENDERNODE13 01m 39s 0.748220 12 453312 KB 3 20-29 Completed RENDERNODE11 02m 46s 1.033258 12 446124 KB 4 30-39 Completed RENDERNODE10 03m 21s 1.251113 12 444532 KB 5 40-49 Completed RENDERNODE12 03m 40s 1.369378 12 430544 KB 6 50-59 Completed RENDERNODE14 02m 18s 1.042973 12 448512 KB 7 60-69 Completed RENDERNODE13 02m 33s 1.156340 12 453156 KB 8 70-79 Completed RENDERNODE11 03m 49s 1.425398 12 491912 KB 9 80-89 Processing RENDERNODE10 1.313358 12 0 KB 10 90-99 Processing RENDERNODE12 1.232440 12 0 KB 11 100-109 Completed RENDERNODE14 02m 18s 1.042973 12 520700 KB 12 110-119 Completed RENDERNODE13 02m 15s 1.020300 12 517204 KB 13 120-129 Processing RENDERNODE14 0.347658 12 0 KB 14 130-139 Processing RENDERNODE13 0.211618 12 0 KB 15 140-149 Processing RENDERNODE11 0.155611 12 0 KB 16 150-159 Queued 12 0 KB 17 160-169 Queued 12 0 KB 18 170-179 Queued 12 0 KB 19 180-189 Queued 12 0 KB 20 190-199 Queued 12 0 KB 21 200-209 Queued 12 0 KB 22 210-219 Queued 12 0 KB 23 220-229 Queued 12 0 KB 24 230-239 Queued 12 0 KB 25 240-249 Queued 12 0 KB 26 250-259 Queued 12 0 KB 27 260-269 Queued 12 0 KB 28 270-279 Queued 12 0 KB 29 280-289 Queued 12 0 KB 30 290-299 Queued 12 0 KB 31 300-300 Queued 12 0 KB
In the job queue view, right-click on one of the rendering jobs and select the "Job Slices..." option from popup menu. This will bring up the job slice view which shows the status of each job slice.
33
SquidNet Network Render Manager
3.16.1.4 Application log
With the Job Slice window still open, right-click on any of the completed jobs slices and select the "Application log..." option from the popup menu. This will open up the application's rendering log which shows the command that SquidNet used to process the request and the output from the application.
3.16.1.5 Render Farm View
Equivalent CLI Command sqn --renderFarmShow Node IP Type O/S State Version Cores CPU Usage Active Processing GHZ/hrs Memory Free Uptime -----------------------------------------------------------------------------------------------------------------------------------------------------------------------THOR 10.0.0.28 MASTER WIN8x64 Idle 2.66P102 12 10 2 0 38.424000 67050720 63360768 11h 55m 56 RENDERNODE10 10.0.0.90 SLAVE VISTA64 Busy 2.66P102 8 95 0 1 22.344000 8345796 5445272 01h 50m 39 RENDERNODE12 10.0.0.92 SLAVE VISTA64 Busy 2.66P102 8 7 0 1 22.344000 8345796 6975236 01h 50m 33 RENDERNODE11 10.0.0.91 SLAVE VISTA64 Busy 2.66P102 8 93 0 1 22.344000 8345796 5361320 01h 50m 21 RENDERNODE13 10.0.0.93 SLAVE VISTA64 Busy 2.66P102 8 64 0 1 27.200000 8055948 6032168 01h 50m 50 RENDERNODE02 10.0.0.82 CLIENT VISTA64 Idle 2.66P102 8 3 0 0 27.200000 8055948 6892580 01h 51m 00 RENDERNODE14 10.0.0.94 SLAVE VISTA64 Busy 2.66P102 8 60 0 1 27.200000 8055948 6208360 01h 50m 58
The Render Farm view shows that status of all render farm nodes. If your submitted jobs are still running, you should see the state of all slave nodes set to "Busy"
34
SquidNet Network Render Manager
3.16.1.6 Farm Work Queue View
Equivalent CLI Command sqn --netQueueShow Node Job QueTime Source Template Jobslice ----------------------------------------------------------------------------------------------------------------------------RENDERNODE10 MAYA-JOB#2 29m 56s THOR Autodesk Maya 8 RENDERNODE11 MAYA-JOB#2 30m 38s THOR Autodesk Maya 7 RENDERNODE12 AFX-JOB#2 21s THOR Adobe After Effects 186 RENDERNODE13 MAYA-JOB#2 09m 04s THOR Autodesk Maya 9 RENDERNODE14 MAYA-JOB#2 01m 17s THOR Autodesk Maya 10
The Farm Work Queue view shows a list of all active render nodes and the jobs that they're currently working on.
3.16.1.7 Image Preview View
The Image Preview view displays the output images from the output folder of any selected job. Click on any job in the project or job queue views and notice how the Image Preview window starts scanning the output folder and displays the selected jobs rendered images. If more than a few images have been rendered, press the "Play" button to play back the image sequence in real-time.
3.17 Video from image sequences This section will show you how to create video files from your image sequences.
35
SquidNet Network Render Manager
3.17.1 Playback application In order to auto-playback rendered videos, set the path to your favorite video player app in the preferences dialog (Admin>Preferences...).
Figure 16 Video Player Path Setting For this tutorial we'll be using the free Video Lan (VLC) player. In this case, the path is set to "C:/Program Files (x86)/VideoLAN/VLC/vlc.exe"
3.17.2 After Effects Video Press the "New Job" button and select the FFMPEG template from the top-left of the job submission form.
Figure 17 After Effects Video Creation 1.
2.
On left-side of the form: a. Set "Job ID" to AFX-VIDEO#1 b. Select the "Create output folder" button. On right-side of the form, set the following fields: a. Frame rate: 24 fps b. Image Sequence Folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/AE c. Video output folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Videos d. Select "Play Video" button.
36
SquidNet Network Render Manager
3. 4.
Press the "Submit" button. When asked to save job, say "yes"
Equivalent CLI Command sqn --submit --jobId AFX-VIDEO#2 --template ffmpeg --appPath ffmpeg --inputDir //RAIDSERVER00/Volume_1/SquidNet/Tutorial-Images/AE --outputDir //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Videos --videoFrameRate "24 fps" --videoResolution "hd720(1280x720)" --useAllCores --videoPlay
3.17.3 Maya Video This time, instead of creating a job from a template, we'lre going to create a new FFMPEG job an existing job profile. In the SQUIDNET-TUTORIAL project, open the AFX-VIDEO#1 job profile created in the previous step and change the following fields:
Figure 18 Maya Video Creation 1.
2.
3. 4.
On left-side of the form: a. Set "Job ID" to MAYA-VIDEO#1 b. Make sure the "Create output folder" checkbox is checked. On right-side of the form, set the following fields: a. Frame rate: 24 fps b. Image Sequence Folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Maya c. Video output folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Videos d. Select "Play Video" button. Press the "Submit" button. When asked to save job, say "yes"
Equivalent CLI Command sqn --submit --jobId MAYA-VIDEO#2 --template ffmpeg --appPath ffmpeg --inputDir //RAIDSERVER00/Volume_1/SquidNet/Tutorial-Images/Maya --outputDir //RAID-SERVER00/Volume_1/SquidNet/TutorialImages/Videos --videoFrameRate "24 fps" --videoResolution "hd720(1280x720)" --useAllCores --videoPlay
37
SquidNet Network Render Manager
3.17.4 Modo Video In the SQUIDNET-TUTORIAL project, open the AFX-VIDEO#1 job profile created in the previous step and change the following fields:
Figure 19 Modo Video Creation 1.
2.
3. 4.
On left-side of the form: a. Set "Job ID" to MODO-VIDEO#1 b. Make sure the "Create output folder" checkbox is checked. On right-side of the form, set the following fields: a. Frame rate: 24 fps b. Image Sequence Folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Maya c. Video output folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Videos d. Select "Play Video" button. Press the "Submit" button. When asked to save job, say "yes"
Equivalent CLI Command sqn --submit --jobId MODO-VIDEO#2 --template ffmpeg --appPath ffmpeg --inputDir //RAIDSERVER00/Volume_1/SquidNet/Tutorial-Images/Modo --outputDir //RAID-SERVER00/Volume_1/SquidNet/TutorialImages/Videos --videoFrameRate "24 fps" --videoResolution "hd720(1280x720)" --useAllCores --videoPlay
3.18 Modo Tile Rendering To render a single frame across multiple render nodes, do the following:
38
SquidNet Network Render Manager
Figure 20 Modo Tile Render Form
Figure 21 Modo Tile Render Setup
39
SquidNet Network Render Manager
Figure 22 Modo Tile Render Viewer 1. 2.
3.
4.
5. 6. 7.
In the SQUIDNET-TUTORIAL project folder, open the MODO-JOB#1 job that was created earlier. On the left-side of the form: a. Change "Job ID" to MODO-JOB-TILE-RENDER#1 b. Make sure the "Create output folder" checkbox is checked. On the right-side of the form, set the following fields: a. Application: $APP(SQUIDNET-TUT-MODO) b. Project File: //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Modo/racecar.lxo c. Start Frame: 1 d. Ending Frame: 1 e. Frame Per Slice: 1 f. Output Image Path: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo-Tile-Render g. Image Prefix: racecar_ h. Output Image Format: png i. Render Camera: OVERHEAD On the left-side of the form: a. Press the "Tile Rendering" button in “Render Options” section. b. In "Tile Rendering Setup" dialog set the following values: i. Resolution Preset: HD 720 ii. Format: png iii. Move row slider to select 8 rows. iv. Move col slider to select 5 columns. v. Press the "Select All" button to select all rows and columns. Alternatively, use the mouse to select specific cells in the grid. vi. Set the "Enable" checkbox. vii. Press "Apply" button. Press the "Submit" button. When asked to save job, say "yes" In Job Queue view, right click on the newly submitted MODO-JOB-TILE-RENDER#1 job and select the "Tile Render Viewer" to see the tile render operation in real-time.
Equivalent CLI Command sqn --submit --sceneName MODO-JOB-TILE-RENDER#2 --template modo --tileRequest --tileRenderer "modo" --tileRows 8 -tileCols 5 --tileMode grid --tileRegions all --tileImagePrefix racecar_ --tileImageFmtIdx 1 --appPath $APP(SQUIDNET-TUTMODO) --sceneFile //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Modo/racecar.lxo --startFrame 1 --endFrame 1 --priority 4 --outputDir //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo-Tile-Render --imagePrefix racecar_$(REG_MASK) -imageWidth 1280 --imageHeight 720 --imageFormat png --useAllCores --include
40
SquidNet Network Render Manager
3.19 Using Skip Frame SquidNet has the ability to ignore the rendering of frames that already exist in the output folder. When this feature is enabled in the job request and submitted to the render queue, SquidNet will scan the output folder and determine which frames are present and to which jobs slices those frames are assigned to. If any frames are missing (i.e; no image files), SquidNet will only process the job slices that contain the missing frames. Do the following to get an idea on how the skip-frame feature works: 1.
2. 3. 4. 5.
Using the modo MODO-JOB#1 job profile that we set up earlier, delete some random images files from the job’s output folder (//RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo). Note the frame numbers of the images that were deleted. In the project view, open the “MODO-JOB#1” job profile and change the name to “MODO-JOB#1 SKIP-FRAME”. Under the “Basic Settings” section, select the “Skip Existing Frames” checkbox. Press the submit job button. Save the job if asked to do so. With the “MODO-JOB#1 SKIP-FRAME” now added to the render queue, open up the job slice view for the job and note how only the slices that contained the frames that were previously removed will be processed. The slices for the frames that were present in the output folder should be marked as “Complete”
Figure 6 Skip Existing Frames
3.20 Using Frame-check Skip-check checks for frames BEFORE a job a is processed. Frame-check checks frames AFTER slices are processed. With framecheck enabled, SquidNet will examine the job slice’s output folder and automatically create pickup-jobs for frames that match the following criteria:
Missing image files. Zero-sized images files. Images that are proportionally smaller than the other images in the same folder. (see preferences for default setting).
Figure 7 Frame Check
41
SquidNet Network Render Manager
Do the following to get an idea on how frame-check works: 1. 2. 3. 4. 5. 6.
Using the modo MODO-JOB#1 job profile that we set up earlier, delete half of the image files in the job’s output folder (//RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo). In the project view, open the “MODO-JOB#1” job profile and change the name to “MODO-JOB#1 FRAME-CHECK”. Under the “Basic Settings” section, select the “Frame Check” checkbox. Press the submit job button. Save the job if asked to do so. While the “MODO-JOB#1 SKIP-FRAME” is in the processing state, remove a few newly created images from the output folder BEFORE the rendering job slices completes. When the job slice completes, SquidNet will create a pickup-up job for any missing frames. You should see some new jobs named “MODO-JOB#1 SKIP-FRAME (pick frames - xxxxxxx)” in the job qeueue view.
3.21 Using Image Display The image-display feature displays preview images of rendered frames on the SquidNet GUI of the submitting workstation.
Figure 8 Image Display To enable image-display, do the following: 1. 2. 3. 4.
Open the “MODO-JOB#1” job profile and change the name to “MODO-JOB#1 IMAGE-DISPLAY”. Under the “Render Options” section, pressed the “Advanced” button. Press the “Configure” button for the Image Display field. In the “Display preview images” field, enter a number that indicates the interval for displayed images. For example, a value of “10” will display a preview image for every 10th frame rendered.
Figure 9 Rendered Image Preview
42
SquidNet Network Render Manager
Chapter 4 INSTALLATION
43
SquidNet Network Render Manager Installation
4. INSTALLATION 4.1 Overview The following is a list of tasks that must be performed to get your SquidNet render farm running: 1.
2.
3. 4.
5.
6. 7.
Prepare network resources: o Ensure there’s enough disk space on each render farm computer. o Ensure there’s enough memory (RAM) on each render farm computer that will be used for rendering. o Setup network share paths (SAN/NSA, etc …) and ensure that the correct access permissions are in place. Install rendering applications on render farm computers. o Each rendering node must have a local copy of each rendering application. Some applications allow the “rendering utility” to be placed on a network share. Check your application’s documentation to see if this is possible. o If required, verify that application render licenses are in place. On Windows, create the SquidNet Service Account (SSA) or use an existing ADMIN account that exists on all Windows-based render farm nodes. Install SquidNet on all render farm nodes using the following procedures: o SquidNet Basics: (open) o SquidNet Installation Guide for Windows: (open) o SquidNet Installation Guide for Linux: (open) o SquidNet Installation Guide for Mac OS: (open) Configure SquidNet render nodes: o Configure master node. Configure SquidNet configuration path on network share folder. Install SquidNet licenses o Configure client nodes. Configure SquidNet configuration path on network share folder Other preferences (Determine which applications you’ll be submitting jobs to, etc…) o Configure slave nodes. Set up application profiles. (i.e.; tell SquidNet where rendering applications are installed on each node) If using a cross-platform configuration, setup translation profiles. (i.e.; tell SquidNet how to use network paths on different operating systems)
All installation documents are available on SquidNet’s website: http://www.squidnetsoftware.com.
4.2 Requirements Here are some basic operating system requirements to consider: WINDOWS hosts:
Use 64-bit operating systems to take full advantage of the larger memory capabilities. SquidNet will work just fine on a 32-bit version will but will be limited to 3Gbytes of addressable memory space (if the operating system is configured properly). Note: In the future, SquidNet will not support 32-bit operating systems. Use WINDOWS SERVER products to avoid the TCP/IP connection limits imposed by Microsoft on non-server O/S versions (WIN8, WIN7, VISTA, WIN-XP and WIN2000). Use of WINDOWS HOME editions is not recommended because of the lack of network connectivity support. Use of a Network Access Storage server (NAS) is highly recommended. NAS devices are relatively easy to configure and provide a better data protection option than WINDOWS resident drives. Consider using a NAS that supports RAID technology to better protect your scene content. Configure virus software to allow SquidNet to access system resources. In most cases, it’s recommended that you disable your virus software until your configuration is complete. Check your virus documentation and SquidNet installation documetion for configuration settings.
Linux hosts:
Use 64-bit operating systems to take full advantage of the larger memory capabilities. SquidNet will work just fine on a 32-bit version will but will be limited to 4Gbytes of addressable memory space. Note: In the future, SquidNet will not support 32-bit operating systems. If using a 64-bit operating system, make sure you install the 32-bit operating system libraries. See Linux installation documentation for additional information.
44
SquidNet Network Render Manager Installation
Use a SAMBA server to allow local disk volumes to be used as network storage devices. If used as a network storage device, make sure to acquire a large capacity disk drive. Lower cost: Most Linux distributions are free (or close to it) so they provide a better-cost alternative to Microsoft’s or Apple’s licensing models. Use the more common Linux distributions like Red Hat, Fedora, Ubuntu, CentOS and OpenSUSE.
Mac OS hosts:
Use iMac-class computers as they have more CPU cores and larger memory configurations.
Consult your hardware and software documentation for optimal performance requirements.
4.2.1 Hardware set up From a hardware perspective, the follow hardware considerations are recommended:
Multi-core computers: Use CPUs that have at least 2 cores. SquidNet does not support single-core render nodes. Lots of RAM: The minimum memory requirement for most 3D rendering applications is 8GB. Make sure to properly size your render farm by making sure that the rendering requirements of your scenes line up with the amount of RAM available on the rendering computers. Storage content: Use network-accessible NAS/SAN content storage. Using Windows shares with render farms that have more than 5-6 nodes is not recommended because of Windows’ network connection limitations. 1GBit network: Use 1GB (or higher) network configurations.
4.2.2 NAS/SAMBA Storage Devices If your system has 10 or more render nodes, it’s strongly recommended that you use either a Network Access Storage (NAS) device or a Linux machine with a configured SAMBA server (A free open source solution can be found at http://www.freenas.org). Using either will keep you from getting Windows System Error #267 errors when too many connection attempts are made to a WINDOWS network drive. (This does not apply if you’re using a WINDOWS Server configuration with the appropriate Microsoft connection licenses.)
4.2.3 Network configuration Some network recommendations:
Use static IP addresses for all nodes. Do not use DHCP addressing because of the likelihood that nodes will change their IP address between restarts. If any node has more than one network interface enabled (multi-interface or multi-subnet configuration), use the Network Address Selector dialog (Configuration->Network Address...) in the Render Farm view to select the network interface that SquidNet will use on that node. By default, SquidNet auto-selects the local network interface. In a cross-platform environment, it may be necessary to update the operating system’s “hosts” files to make sure that all nodes can “see” each other. On Windows, add Linux and Mac OS hostnames to the C:\Windows\System32\drivers\etc\hosts file. On Linux and Mac OS, add entries to each node's /etc/hosts and /private/etc/hosts files, respectively. Make sure that the SSA account has full access to all network shared resources (NAS, Mapped drives, etc...). Verify access by logging into a node using the SSA account credentials and attempt to create/delete files from network share folders. Using WIFI-based network adapters is not recommended. Always use direct-wired network interfaces.
4.2.3.1 Setting up network shares The following sections will describe setting up network shares.
4.2.3.1.1 WINDOWS Windows provides 2 methods for accessing network shares:
Mapped Drives: Mapped drives are paths that are assigned a specific driver letter and to the user appear as regular hard drives (D:\, E:\, etc…)
45
SquidNet Network Render Manager Installation
UNC Paths: UNC paths are absolute paths to network shared folders and look like this: \\raidserver00\volume_1\SquidNet (for example). Mapped drives can optionally point to UNC paths.
Using either is basically a matter of preference. However, UNC paths are easier to use because their destination paths are more easily distinguishable whereas with mapped-drives, their actual location is hidden behind the drive letter. Note: When using mapped-drives, SquidNet will always mirror the submitting workstation’s mapped-drives on the rendering nodes. For example: If a submitting workstation has drive X:\ mapped to \\NASBOX\Projects, each rendering node will recreate the same mapping before the rendering process begins. This ensures that any internal scene references to mapped drives will still be valid. Also, mapped drives are session-based --- meaning that they’re only mapped (active) while the rendering process is running. As soon as the rendering session is closed, the mapped drives are “unmapped”. SquidNet never mounts (or unmounts) mapped drives on local user sessions. To create a mapped drive “Z:\” that points to network folder “\\raid-server00\volume_1\SquidNet” use the following command in a DOS shell (alternatively, use the Mapped Drive feature on Windows’ File Explorer):
net use z:\ \\raid-server00\volume_1\SquidNet
4.2.3.1.2 Mac OS To mount a network shared folder, open a terminal and execute the following commands:
sudo mkdir /Volumes/SquidNet sudo mount -t smbfs //raid-server00/Volume_1/SquidNet /Volumes/ SquidNet
The above command assumes that the network folder is \\raid-server00\volume_1\SquidNet (from a Window’s perspective)
4.2.3.1.3 Linux To mount a network shared folder, login as root and execute the following command:
sudo mount -t cifs -o guest //raid-server00/volume_1/SquidNet /mnt/SquidNet/
The above command assumes that the network folder is \\raid-server00\volume_1\SquidNet (as seen from a Window’s host) To make sure that the mount point persists between reboots, edit the /etc/fstab file to include:
//raid-server00/volume_1/SquidNet /mnt/SquidNet/ cifs guest,_netdev 0 0
4.3 Operating System Accounts 4.3.1 Windows SquidNet must run under privileged accounts that have full access to local and network resources. On Windows, the SquidNet service runs under a workgroup administration account called the SquidNet Service Account (SSA). The SSA account is important because it allows SquidNet to access local and network resources that have been configured to allow access. (via specific account name and password settings). Windows will not allow access to certain resources if the account credentials are not the same. When the Windows SquidNet installer is executed you’ll be prompted for the SSA account name and password. It’s advisable to have a dedicated SSA with full privileges on each node that SquidNet will be installed on. Most users create a special “squidnet” or “render” admin account (although, any account name will suffice) to act as the SSA. Here’s some SSA prerequisites and notes: 1. 2. 3.
4.
Make sure to use the same SSA account name and password exists on all Windows hosts. Doing so will prevent you from having access permission issues when certain nodes try accessing network shares. On WINDOWS SERVER nodes, when a new user account is created, the account is NOT part of the Administrators Group (only part of Users Group) so make sure to add the SSA account to the Administrators Group. Make sure that the SSA account has full read/write/access permissions on all network resources and to add the SSA account to access control lists on all network share folders. In order for WINDOWS machines to access each other's network content the correct security settings must be in place beforehand. If accessing network share content on a WINDOWS server machine (WIN2008, WIN2012, etc...), all network nodes accessing the server MUST use the same SSA account used during the SquidNet installation procedure.
46
SquidNet Network Render Manager Installation
Account Type Local Account Domain Account
Service Logon Type Local Account Local System
Domain Local node name Domain controller
User
Password
Figure 10 Windows SquidNet Service Configuration On domain controller networks, make sure that the SSA account (which resides on the domain controller (PDC)) is also a member of the Local Administrators Group on every render farm node. To add the SSA to Local Administrators Group, do the following:
At the "Start" menu, type "mmc" into search box. If "Local Users and Groups" folder is missing: o Select File o Select "AddRemove Snap-in" o Select "Local Users and Groups" and then click add. o Click "Local Computer" and then click Finish. Double-click Groups folder Double-click Administrators. Add the SSA account to Administrators group.
Figure 11 Local Administrators Group Depending on your security policies, it may be necessary to run the SquidNet service under the Local System account. To do so, use the "Change Log On account (Windows only)" option in the Render Farm view to choose between using the Local System account or SSA.
Figure 4 Window Change Log On Account
47
SquidNet Network Render Manager Installation
Figure 5 Windows Local System and SSA accounts
4.3.2 Mac OS and Linux On Mac OS and Linux, SquidNet must be installed and run under the system “root” account.
4.4 Operating System preparation 4.4.1 Windows For Windows platforms, do the following prior to installing SquidNet:
Make sure you have admin rights to host. Disable User Account Control (UAC) Settings. Completely disable firewall or enable the following ports: o TCP Port 17394 o UCP Port 17395-17400 o Note: the SquidNet service will attempt to create these firewall profiles on startup. Create a dedicated SquidNet Service Account (SSA): o Must be an administrator account. o Use SSA credentials when installing SquidNet on each Windows computer. o Name SSA account accordingly. Typical account names are “render”, “rendermanager” or “squidnet” o Make sure the same SSA account and password is used on all nodes. If using domain controller: o Make sure to assign network access permissions to SSA account. o Make sure to added SSA account to Local Administrators Group on every render node. To test SSA account, log in on to any Windows computer and make sure you can access all network resources (shares, mapped drives, etc…) by creating and deleting some temporary files. If using a cross-platform render farm, update Windows’ hosts file (C:\Windows\System32\drivers\etc\hosts) with the IP addresses and node names of Mac OS or Linux nodes.
4.4.2 Mac OS For Mac OS platforms, do the following prior to installing SquidNet:
Make sure you have sudo/root access. Make sure you have enough disk space. Make sure to change the default Mac OS name from “localhost” to something more meaningful. To change, use the following commands: o Change: sudo scutil --set HostName o Check: sudo scutil --get HostName If using a cross-platform render farm, update Mac OS’s hosts file (/private/etc/hosts) with the IP addresses and node names of WIndows or Linux nodes.
4.4.3 Linux For Linux platforms, do the following prior to installing SquidNet:
Make sure you have sudo/root access.
48
SquidNet Network Render Manager Installation
Make sure you have enough disk space. If using a cross-platform render farm, update your operating systems hosts file (/etc/hosts) with the IP addresses and node names of Windows or Mac OS nodes.
4.5 Installing SquidNet SquidNet runs as a background process on all operating systems that it supports. On Windows, background processes are called Services while on Mac OS and Linux, they’re called daemons (also called services in some cases). A background process runs behind the scenes and doesn’t have a visual component (GUI or console interface). The SquidNet service runs whether or not any user is actively logged onto the host computer. The table below lists the name of the SquidNet service that runs on each platform: Operating System Windows Mac OS
Service (daemon) name squidserver.exe squidnetd
Linux
squidnetd
Check running status Use Windows Task manager Use “top” command from terminal shell or Activity Monitor Use “top” command
Table 3 SquidNet Process Name Install SquidNet on each render farm host per the following installation instructions:
SquidNet Installation Guide for Windows. (open) SquidNet Installation Guide for Linux. (open) SquidNet Installation Guide for Mac OS. (open)
Once each node is installed, verify that the installation was successful by issuing the following command: CLI command: Verify that SquidNet service is running. All platforms sqn –status
4.5.1 Starting and Stopping If it’s ever necessary to start/stop or restart the SquidNet server, you can do so by using the sqn CLI program. On Mac OS and Linux platforms, access so “sqn” is globally accessible (via a soft link in /usr/local/bin folder). On Windows, you’ll need to modify your system path variable (PATH) to include the SquidNet installation directory. Task start stop restart
sqn sqn --start sqn --stop sqn --restart
Windows net start squidserver net stop squidserver net stop squidserver net start squidserver
Mac OS sudo launchctl start com.squidnet.server sudo launchctl stop com.squidnet.server sudo launchctl stop com.squidnet.server sudo launchctl start com.squidnet.server
Linux sudo /sbin/service squidnet start sudo /sbin/service squidnet stop sudo /sbin/service squidnet restart
Table 4 SquidNet Service Management
4.6 Node configuration 4.6.1 Master host location Where should the master host reside? Well, that depends on your workflow. The master, if not configured for rendering jobs, is basically a message router and doesn’t need a lot of operating system resources. At a minimum, the host that the master will run on must adhere to the following requirements:
4GB of RAM: More if configured as a rendering node or for large numbers of queued jobs. 2 CPU cores: The master is heavily multi-threaded so the number of cores needed depends on your workflow requirements. However, for optimal performance, the more cores the better! 2GB of free disk-space: Use SSD drive for improved disk performance. 1GB Network interface: 10GB is preferred for heavy traffic.
The master typically does NOT need a dedicated host. In most cases, the master will run fine on a client workstation. If may be necessary to experiment a little to determine the optimal master host location.
49
SquidNet Network Render Manager Installation The master can be configured to render jobs. However, enabling this feature can be problematic if rendering “heavy” scenes that might take up a lot of system resources (CPU, memory, network I/O, etc…). If all resources are used up, the master might get “starved” of CPU and memory resources, which in turn, could result in reduced workflow efficiency. In some cases, the master could become totally unresponsive to any outside access and basically halt all render farm operations. So the bottom line, make sure you understand your workflow requirements before configuring the master to render jobs. To enable the Master to render jobs, select the Master node in the “Render Farm Nodes” view window, right-click, “Configuration->Configure Master…” and select the “Master is allowed to process jobs” checkbox.
Figure 12 Configure master to render jobs
4.6.2 Setup node types Assuming that you know the location of your master node (don’t worry you can always change it later), you’ll need to configure each render farm node to act as one of the following SquidNet node types: SquidNet Node Type Master Client Slave Peer
Tasks The main render farm controller. Manages all network activity. Submits jobs to render farm. Can optionally be configured to render jobs. Render farm “engine”. Its sole job is to render job requests. Un-configured node. Default node type on initial installation.
Table 5 SquidNet Node Types By default, all render farm nodes are set up as PEERS during the initial software installation. PEER nodes don’t do anything. They’re basically in an orphaned state waiting to be configured to either a master, client or slave node type. All nodes in the farm, must be configured to be one of the above listed node types. Configuration must be done in the following order: 1. 2. 3.
Configure the master node first: a. Determine master behavior by enabling specific features. Configure client nodes: a. Determine if, and how, client nodes will render job requests. Configure slaves.
Once all farm nodes are configured, there should never be a need to reconfigure because all nodes “remember” their node types between server restarts. If there’s ever a need to reconfigure your farm, do so by doing the following:
50
SquidNet Network Render Manager Installation 1.
2.
On the existing master node’s GUI (in order): a. Convert all slaves to PEERS. b. Convert all clients to PEERS. c. Convert the master node to a PEER. d. Alternatively, use the “sqn –resetFarm” CLI command to turn all nodes (including master) to peers. On the new master node’s GUI (in order): a. Configure new master node. b. Configure client nodes. c. Configure slave nodes.
Note: When reconfiguring render farm nodes, all node types must be set to PEER beforehand.
4.6.2.1 Setting up master Before configuring your render farm, consider the following:
Select a single node on your render farm to act as a master (controller). The master node should be a computer that will be running most of the time. A reliable user workstation or main server are always good candidates. The master can be configured to optionally render jobs. However, this is not recommended because heavy scenes may bogdown workflow operations. Determine which nodes will act as client nodes. Client nodes are user workstations that submit jobs to the render farm queue. Clients nodes can be configured to process jobs using one of the following methods: a. Never: Client can only submit jobs. It can never process (render) jobs. b. When user is logged out: Client can only process jobs when no users are actively logged in. c. When sceensaver is active: Client can only render jobs when screensaver is running (Mac OS only) d. Low priority: Process jobs at any time but only process them at the lowest possible CPU priority level. Depending on a scene complexity, low-priority rendering may affect user desktop performance. Determine which nodes will act as slave nodes. Slave nodes are the work horses of the render farm and are controlled by the master. Slave nodes can never submit jobs. If master node HAS NOT been defined yet, do the following: 1. 2. 3.
Open the local SquidNet User Interface (GUI) on the computer that is to be set up as the master. Open the Render Farm View window from the main menu (Network->Render Farm...) Select the MASTER node name (should the computer that you’re physically logged on to), right click and select "Configuration->Set As Master…" menu option. Wait for node type to change from PEER to MASTER.
If a master node HAS already been configured, do the following to register a node to the master: 1. 2. 3. 4.
Go to SquidNet User Interface on master computer. Open Network View window from main menu (Network->Render Farm...) Select the new computer's node name and configure accordingly using "Configuration" pop-up menu. Optionally, if logged onto a slave or client workstation, open “Connect to Master as…” dialog (menu: Network>Connect To Master As …) to connect to master remotely.
Figure 13 Connect To Master Dialog. Accessible on client and slave nodes.
51
SquidNet Network Render Manager Installation
Figure 14 Possible SquidNet Render Farm node types. Important: Node configuration CAN ONLY be done on the master's user interface.
4.6.3 Example Configuration
Figure 15 Initial Configuration The image above shows what a typical render farm configuration might look like when SquidNet is installed on the computers listed. Notice how all nodes are initially installed as node type “PEER” (the default). For this example, we're going to assign the appropriate node types to each computer. For the master, make sure that you’re physically in front of the master computer’s monitor. Open the SquidNet User Interface on node "MASTER" and then open the Network View window (Network->Render Farm... menu) and change the node type from peer to master:
52
SquidNet Network Render Manager Installation
Figure 16 Setting up node as SquidNet Master Equivalent CLI Command sqn --masterSet MASTER
Description Sets node MASTER to be the SquidNet master
While still in front of the SquidNet UI on the master node, configure node "CLIENT-001" to be a SquidNet client:
Figure 17 Setting up node as SquidNet Client Equivalent CLI Command sqn --nodeTypeSet client --node CLIENT-001
Description Sets node CLIENT-001 to be a SquidNet client
Likewise, do the same for slaves "SLAVE-001" thru "SLAVE-007" by selecting all the slave nodes at the same time:
Figure 18 Setting up node as a SquidNet Slave Equivalent CLI Command sqn --nodeTypeSet slave--node SLAVE-001, SLAVE002, SLAVE-003, SLAVE-004, SLAVE-005, SLAVE006, SLAVE-007,
Description Sets nodes SLAVE -001 thru SLAVE -007 to be a SquidNet slaves
53
SquidNet Network Render Manager Installation
When complete, the render farm configuration should look like this:
Figure 19 Configured Render Farm
4.6.4 Changing Node Types To change a node type from a client to a slave:
From master UI: Select node change node type to slave. From local client UI: Use "Network->Connect to Master as..." dialog from network main menu.
To change a node type from a slave to a client:
From master UI: Select node and change node type to client. From local slave UI: Use "Network->Connect to Master as..." dialog from network main menu.
To relocate the master to a different node, do the following:
On the existing master UI (physically in front of): o Convert all clients and slaves to type PEER. o Convert the current master node into a PEER node. From the new master UI (physically in front of):: o Select the new master node and change node type to MASTER. o Convert the remaining PEER nodes to either clients of slaves.
4.6.5 Adding nodes from external subnets Upon installation, SquidNet will automatically detect all nodes that are on the same network subnet. This is the default behavior. However, on render farms that are spread across multiple subnets, you’ll need to use the master’s “Add Node” dialog to configure in any external client or slave computers. To access, on the master UI, select the master node name from the list and open the “Add Render Farm node” dialog (“Configuration->Add Node …)”.
54
SquidNet Network Render Manager Installation
Figure 20 Add Node Dialog With the “Add Render Farm Node” dialog open, select the node name from the list or enter the node name or IP address then press the “Lookup” button. If the node is found, determine if it’s supposed to be a client or slave and press the “Apply” button. At this point, the newly added node should appear in the Render Farm Nodes view.
4.6.6 Master Configuration
Client processing mode Client node list Client job permissions
Client admin permissions
Client cores to use
Farm configuration path
Job distribution algorithms Master Enable Processing Job queue notifications
Figure 21 Master Configuration Dialog Once the master, client and slave nodes have been configured, no additional configuration is required to get your SquidNet render farm functional. However, if your farm requires a little more management, you can use the SquidNet Master Configuration dialog to do the following:
55
SquidNet Network Render Manager Installation
Configure client workstation functionality: o Assign workstation rendering mode (render when user is logged off, render at a low CPU priority, never render) o Set a fixed job submission priority for each client workstation. o Assign client workstation to a specific job pool. o Enable/disable specific admin rights: Full admin rights (same as master controller). Ability to create new projects Add new application paths Modify local user preferences Modify existing job pools. Determine how many CPU cores to allocate for job processing. Modify other user’s job submissions. Apply SquidNet configuration path to all nodes. Determine if master node can render jobs (or not). Select job distribution algorithm to use for jobs at the same queue priority. Enable periodic SMS text notifications on job queue status.
To access the Master Configuration dialog, select master node in network window, right-click and select “Master Configuration…”
4.6.6.1 Queuing methods SquidNet provides multiple job distribution algorithms. These algorithms apply to all jobs that reside in the same priority queue. The default is FIFO (First-In-First-Out) which means that the master will sort the job queue based on the oldest job in the queue. Other distribution algorithms are listed below. Distribution Algorithm FIFO LIFO ROUND-ROBIN RANDOM BALANCE-SLICES BALANCE-FRAMES FASTEST SLOWEST FEWEST-SLICES MOST-SLICES FEWEST-FRAMES MOST-FRAMES FEWEST-GHZhrs MOST-GHZhrs
Description First-in-first-out (default) Last-in-first-out Sequentially access job slices Random selection Maintain an even percentage of completed job slices Maintain an even percentage of completed frames Quickest job slice average render times Longest job slice average render times Least amount of job slices in job Most amount of job slices in job Least amount of frames in job Most amount of frames in job Lowest average GHZ hrs used per job slice Greatest average GHZ hrs used per job slice
Table 6 Distribution Algorithms Configure the queuing mething used by the master in the Master Configuration dialog.
4.6.7 Client Configuration SquidNet client nodes can have different operating modes. Based on you unique workflow requirements, select one of the following options: Client Mode Never Logged out Background Screensaver
Description Prevents node from rendering jobs. (ie; submit-only mode) Only render jobs when all users are physically logged out. Render when user is logged in but render at a very low CPU processing priority. Only render jobs when screen saver is running. (Mac OS only)
Table 7 SquidNet Client modes
56
SquidNet Network Render Manager Installation
Figure 22 Client Configuration Dialog CLI command: Setting up client to render job at a low process priority All platforms sqn --clientModeSet background --node RENDERNODE10
To change client processing modes, select one or more client nodes in the Render Farm Nodes view, right-click and select “Configuration->Configure Client…” option.
4.6.8 Slave Configuration Once the slaves have been configured from master UI, no further configuration is required.
4.7 License Registration
Figure 23 License Manager Use the license manager to register and manage SquidNet licenses:
Open GUI on master node. Open License Manager dialog (Help->License Manager...) Press the "Register New License" button and cut/paste in your SquidNet licenses (one at a time). If you’ve been using and evaluation license, you’ll need to remove it (right-click “Remove…”) before registering a new one.
4.8 Setting preferences 4.8.1 Configuration Path By default, the SquidNet GUI stores all user configuration settings in the /settings folder. Configuration settings include:
Project folders and job profiles
57
SquidNet Network Render Manager Installation
Path translations Application profiles UI layouts Pool configurations Email/SMS settings etc...
It's recommended that the default configuration path be changed to an alternate location ----- preferably a network share folder. If users will be submitting jobs from multiple client workstations, the same path should be set in the GUI on each client node. To change the default configuration path, open the Preferences dialog from the Admin main menu bar (Admin>Preferences…):
Figure 24 Configuration Path With the dialog open, browse to the network folder and select the location of the configuration path. Press the “Apply” button to set the path. Do the same on the master node and on each client workstation. The configuration path is not used on slave nodes.
4.8.2 Job Templates
Figure 25 Job Templates Use the job configuration panel to tell SquidNet which applications you're intending on using. These are the job types that will be available from the job submission form.
4.8.3 Notifications SquidNet sends job notifications to users to let them know when specific workflow events occur. Use the notifications settings in the preferences dialog to determine which notifications types you’d like to receive. To view user notifications, select the “Notifications” button at the bottom of the main GUI. Note: button is hidden until at least one notification is received.
58
SquidNet Network Render Manager Installation
Figure 26 Job Status Notifications
4.9 Rendering Application Installations Before you can submit any job requests, SquidNet has to know which applications to send the job to and where that application is installed on each render node. When a job is set up for submission, you must first decide what type of job you want to submit and let SquidNet know how to find that application’s rendering program. On the job submission form there’s an “Application” field where you specify the application that you’d like to use to render the job. This “Application” field can have the one of the following values: Application value Absolute path
Rendering application
Example C:\Programe Files\Autodesk\maya2012\bin\render.exe
render.exe
Pros - Works fine if all applications are installed in the same path on every render node.
Cons - Forces applications to be installed in the same path on every node.
- Works great if you only use one application.
- Every job profile must be updated if a new version of the application is installed. - Rendering application path must be added to system PATH variable on every rendering node.
- Easy to use if rendering application paths are already listed in system’s PATH variable on every render node.
- Requires manual modification of PATH variable for every installed application and for every version of each application. Application Profile (recommended)
$APP(MAYA-2013)
- Allows for a different profile for multiple versions of the same application. - Allows the mixing of different application versions in the same profile.
59
- A separate application profile must be setup for every rendering application.
SquidNet Network Render Manager Installation - Allows use of “all” option to indicate that all applications will be installed in the same path.
Figure 27 Application Path Options As shown in the table above, using application profiles is the recommended ways of letting SquidNet know where applications are installed on the farm.
4.9.1 Application Path Profiles Application Path profiles have the following characteristics:
Allow the grouping of application paths under a specific profile name. Creation of application profiles for specific versions of rendering and compositing applications. Allow application profile entries that have different paths on different render nodes.
The following table shows what some typical application profiles might look like: Application Profile MAYA-2013
Render Node
Renderer
Path
Description
render.exe
C:\Program Files\Autodesk\Maya-2013\bin
MAYA-2012
RENDERNODE01 (WIN)
render.exe
C:\Program Files\Autodesk\Maya-2012\bin
render.exe
D:\Program Files\Autodesk\Maya-2013\bin
MAYA-2012
Render
/opt/autodesk/maya/bin
MODO-701
WIN-RENDER_01 (WIN)
modo_cl.exe
C:\Program Files\Luxology\modo701\bin
LINUX-RENDER-01 (LINUX)
modo_cl
/Applications/Modo/…./MacOS
MAC-RENDER-01 (OSX)
modo_cl
/opt/modo
Assumes all versions of render.exe will be installed in …\bin folder. Node RENDERNODE01 has render.exe installed in specified path Assumes all Windows nodes have render.exe installed in the specified path. Assumes all Linx nodes have Render installed in specified folder. WIN-RENDER_01 has modo_cl.exe installed in specified path. LINUX -RENDER_01 has modo_cl installed in specified path. MAC -RENDER_01 has modo_cl installed in specified path.
Table 8 Application Profile Examples As shown in table above, applications profiles are just lists of nodes and the locations of specific application paths. You can specify the following node types:
: Use the specified application path on all render nodes. Should only be used when all nodes are the same platform AND all nodes have the same application installation path. : The application path only applies to Windows render nodes. : The application path only applies to Mac OS render nodes. : The application path only applies to Linux render nodes. : The application path is only valid for the specified node name.
When a rendering node gets a request to process a job it examines the application path and does the following:
Determines if it’s an absolute path (example: C:\Program Files\....\maya\bin\render.exe”. If so, it verifies that the file exists. If it’s only the applications name (example: render.exe) it scans the PATH environment variable to determine if it can be located. Determine if it’s an application profile (ex: $APP(xxxx). If so, it attempts to extract the path as follows: o Checks if any node names in profile match the local node name (example: RENDERNODE01 C:\Program Files\maya\bin\render.exe). If so, it uses the path mentioned. o Checks if any of the operating system wild card tags match the local operating system (example: , or .) If there’s a match, use the path specified.
60
SquidNet Network Render Manager Installation o
Checks if the wild card tag is present (example: ). If found, use the path specified.
Figure 28 Application Profile Assignments The figure above show the setup of 3 typical application profiles:
Maya: o
The tag tells SquidNet to look for “render.exe” in the path specified on all windows nodes. o The tag tells SquidNet to look for “Render” in the path specified on all Mac OS nodes. o The tag tells SquidNet to look for “Render” in the path specified on all Linux nodes. After Effects: o The tag tells SquidNet to look for “aerender.exe” in the path specified on all windows nodes. o The tag tells SquidNet to look for “aerender” in the path specified on all Mac OS nodes. o tag tells SquidNet to look for “aerender” in the path specified on node OSX-SLAVE-O2 only. This points to the “aerender” renderer that comes with Adobe CS5. o tag tells SquidNet to look for “aerender” in the path specified on node OSX-SLAVE-O4 only. This points to the “aerender” renderer that comes with Adobe CS6. V-Ray: o The tag tells SquidNet to look for “vray.exe” in the path specified on all windows nodes. o The tag tells SquidNet to look for “vray” in the path specified on all Mac OS nodes. o tag tells SquidNet to look for “vray” in the path specified on node OSX-SLAVE-O2 only.
61
SquidNet Network Render Manager Installation
4.9.2 Using Application Path Manager
Figure 29 Application Path Manager The Application Path Manager is what you’ll use to create application profiles that tell SquidNet how to locate applications on your farm. The left side of the window above shows a list of already created profiles and the list on the right shows the node and paths assigned to the selected profile. To use the Application Path Manager do the following:
Open the Application Path Manager dialog (Admin->Path Configuration->Application Path Manager ...) Press the “Create New Profile” button. Select the application template that you’re creating the profile for. Select the operating version that the application is installed on. In the “Render Node” drop-down list, select where the application is installed. Set the absolute path to the application (minus the name of the application). Press the “Validate” button to make sure the path is correct. Press the “Ok” button.
Figure 19 After Effects Application Profile
62
SquidNet Network Render Manager Installation
4.9.3 Using Application Path Scanner
Figure 20 Application Path Scanner The Application Path scanner allows you to scan your entire farm looking for installations of the applications selected. When the scan completes, you’ll be shown a list of all applications located. From this list, you can create new application profiles or add the applications to existing profiles. Note: Depending on the size of your render farm, a full network scan can take a very long time. To use the Application Path Scanner, do the following:
Open the Application Path Manager dialog (Admin->Path Configuration->Application Path Scanner ...) Select the applications you want to search for. At the top of the dialog, select which folders you want to scan and which operating systems to include in the search. Press the “Scan render farm” button. When the scan completes (depending on your farm size, it could take hours to complete) select the application found and either create an new profile for it or apply it to an existing one.
4.9.4 Using Application Profiles Once you’ve defined the application profiles for the applications that you’d like to use, you can use the profile in a job submission request for both the GUI or CLI interfaces. To use an application profile in the GUI, select your job template and then add an $APP(xxxx) macro to the “application” field. The “xxxx” will be the name of your profile (example: $APP(MAYA-2013) ).
Figure 30 Using $APP(MACRO) 63
SquidNet Network Render Manager Installation Equivalent CLI Command sqn --submit --jobId MODO-JOB#2 --template modo --appPath $APP(SQUIDNET-TUT-MODO) --sceneFile //RAIDSERVER00/Volume_1/SquidNet/Tutorials/Modo/racecar.lxo --startFrame 1 --endFrame 300 --framePerSlice 10 --priority 12 -outputDir //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo --imagePrefix racecar_ --imageFormat png -useAllCores --createOutputDir --camera OVERHEAD
4.10 Translation Path Profiles
Figure 31 Translation Paths Translation paths allow SquidNet to map the same physical network share folder between different operating systems. The syntactically different network paths (as seen by each operating) system are grouped together inside a translation profile. When submitting cross-platform job requests, you specify the path to use by encapsulating the translation profile name in the $XPATH() macro. Use the $XPATH(xxx) macro in GUI template fields and in CLI job requests.
4.10.1 Translation Path Manager In the mixed (cross-platform) render farm environments (WINDOWS, LINUX, Mac OS), path translations must be registered using the Path Translation Manager to let SquidNet know how to "map" different logical network paths to the same physical path.
Figure 32 Translation Path Manager
64
SquidNet Network Render Manager Installation The figure above shows the Path Translation Manager. As you can see, there's a few path translation profiles that show the physical paths on WINDOWS, LINUX and Mac OS computers and how they point to the same logical drive on the network. For example: The Translation profile MAYA_PROJECTS maps the following paths on each machine to the same network folder:
WINDOWS: \\raid-server00\volume_1\SquidNet\MayaProjects LINUX: /mnt/raid/MayaProjects Mac OS: /Volumes/Volume_1/SquidNet/MayaProjects
In a mixed operating system environment, use path translations to allow SquidNet to properly map job requests between different operating system paths. In your job request, just use the $XPATH() macro to refer to a specific translation profile (example: $XPATH(MAYA_PROJECTS), $XPATH(LIGHTWAVE_PROJECTS), etc...)
4.10.2 Using Translation Paths
Figure 33 Using $APP() and $XPATH() variables The figure above shows a cross-platform job using the $APP() and $XPATH() macros. In the example shown, the macros translate as follows:
$APP(MAYA-2011) maps to the following applications (as declared in the Application Path Manager): o Mac OS Maya command line renderer application (Render) installed on /Applications/Autodesk/maya2011/Maya.app/Contents/MacOS folder on a specific Mac OS based node. o WINDOWS command line renderer application (render.exe) installed on C:\Program Files\Autodesk\Maya2011\bin on a specific WINDOWS based computer. $XPATH(SQUIDDIR) maps to the following paths (all point to same location on the network): o “\\raid-server00\volume_1\SquidNet“ on a WINDOWS-based machine. o “/mnt/raid/SquidNet“ on a LINUX-based machine. o “/Volumes/Volume_1/SquidNet“ on an OS X based machine.
Note: You do not need translation profiles if your render farm has the same type of computers (all Windows, all Mac OS or all Linux). However, you could use translation profiles to set up specific macros that you’d like to use often; For example, you can use the translation macro $XPATH(IMAGE-DIR) in any GUI or CLI field to refer to a specific network folder.
4.11 Upgrading 4.11.1 Remote installation using user interface Once the initial installation is done on each node, SquidNet makes updating your render farm easy by providing the following options:
Graphical User Interface:
65
SquidNet Network Render Manager Installation
o Update entire farm by selecting “Update farm” menu option from Network View window. o Update selected nodes by selecting “Update node” menu option from Network View window. Command Line Interface: o Use “--updateFarm” option to update entire farm. o Use “--updateNode” option to update selected nodes.
Figure 34 Update Render Farm Option
CLI command: Upgrading all nodes on render farm with one command Windows sqn –updateFarm C:\SquidNet-releases\latest Mac OS or Linux sqn –updateFarm /opt/SquidNet-releases/latest
Table 9 CLI: Updating render farm To use “--updateFarm” CLI just point to the path that contains the SquidNet installer files. See the command line interface for additional CLI options.
66
SquidNet Network Render Manager
Chapter 5
USER INTERFACE OVERVIEW
67
SquidNet Network Render Manager
5. USER INTERFACE OVERVIEW SquidNet User Interface Menu bar Main menu
Render Farm Usage
Client Area
Job Queue Completion Local node name Exception List Notifications
Figure 35 SquidNet User Interface The image above shows the main SquidNet user interface. It consists of the following key areas:
Client area: This is the part of the GUI where window “views” can be arranged. The windows can be the job queue, render farm and project profiles views (among others). Main menu: Dropdown menus for all management operations. Menu bar: Quick-access buttons to menu items found in the main menu. Notifications: Status messages for submitted jobs on the local computer. Exception list: Render farm errors and status messages. Job queue completion: Indicator showing completion status of render farm job queue. Render farm usage: Indicator showing percentage of render farm being utilized. Local node name: Hostname of local computer.
Some of the key features of the GUI are:
Multiple view arrangements: Organized the views in the main client area in any configuration layout ( Window>Layouts ...). Tabbed View: Expand the size of each view window to fill the client area and select each window from a tab at the top of the client area (Windows->Tabbed View ...). Full screen: Maximize the size of the GUI to fit your monitor (View->Fullscreen...). Persistent view layouts: The current view layout is saved to disk when the GUI is closed and fully restored when the GUI is restarted. Saved Layouts: Name, save and restore different layouts (Windows->Layouts …).
The GUI has 2 types of windows:
Dialogs: “popup” windows that are displayed on top of the main user interface. Most of these windows are temporarily used to set up specific operating conditions. All SquidNet “manager” windows are dialog-based. Views (client area sub-windows): Windows that only reside within the main GUI client area. The client area can hold any number of windows views and they can be arranged in cascading, tiled, tabbed or full screen orientations. Arrange view windows to match your layout preferences.
68
SquidNet Network Render Manager
Figure 36 Main Window Views The figure above shows the main SquidNet GUI with the Project, Render Farm, Job Queue, Farm Work Queue and Image Preview views.
5.1 License Manager
Figure 37 License Manager The License Manager is used to manage you SquidNet licenses (Help->License Manager ...). Use the license manager to do the following:
View the status of your existing SquidNet licenses. Register evaluation or purchased licenses. Display render farm hardware signatures.
69
SquidNet Network Render Manager
5.1.1 Registering Licenses
Figure 38 License Registration Dialog
To register a SquidNet license, press the “Register New License” button to open the License Registration dialog. With the License Registration dialog open, simply cut/paste in your license then press the “Register” button. You only need to register the license on the master node and it only has to be done once. If the master node is ever relocated, you may need to re-register your SquidNet licenses.
5.1.2 Hardware Signatures
70
SquidNet Network Render Manager
Figure 39 Render Farm Hardware Signatures The figure above shows the Render Farm Hardware Signatures dialog. It display a list of each render node on your farm with a unique hardware id that identifies the hardware devices installed on that node. When purchasing SquidNet license you’ll need to forward us your hardware keys before any official licenses can be issued.
5.2 Preferences The Preferences dialog (Help->License Manager ...) allows you to set specific user and job request settings. It contains the following panels:
Configuration: Local configuration settings. Job Submission: Setting that are specific to job requests. Notifications: Settings for receiving notifications for specific job events.
5.2.1 Configuration Settings
Figure 40 Preferences – Configuration
71
SquidNet Network Render Manager
The configuration panel in the job preferences dialog displays the following settings:
Configuration path: The network share folder where all GUI settings are saved to. In a multi-user environment, make sure that all clients (and the master) point to the same network folder. Splash screen: Enable or disable the SquidNet splash screen when the GUI starts up. Toolbar Appearance: Determines how the main menu tool bar icons are displayed. Power Management: o Option to send WOL packets to render farm nodes when jobs are submitted from the GUI/CLI. o Option to automatically shutdown clients or slaves when they’ve been inactive for a specific period of time. Page Size: Sets the default page size for the job and slice views. Video Player: Video player application that’s used to playback rendered video files.
5.2.2 Job Submission Settings
Figure 41 Preferences - Job Submission The job submission panel displays the following settings:
Lists of supported and configured application templates. Job sound alerts: Play sound clips for specific job events. Framecheck requeue size: Specifies the minimum size percentage criteria for frames that need to be requeued because their size was drastically different than other images in the same output folder. Jobslice Stall Limit: Automatically requeue job slices if they appear “stuck” after a specific period of time. The stall time is determined by inactivity in the application’s output log file.
72
SquidNet Network Render Manager
5.2.3 Notifications Settings
Figure 42 Preferences - Event Notifications Notifications notify the user of specific job processing events. Select the notifications that you’re interested in receiving. When notifications are available, you’ll see a notifications button appear at the bottom of the GUI.
Figure 43 Event Notifications (Green button on bottom) The notifications button is not visible if there are no events present.
73
SquidNet Network Render Manager
Figure 44 Event Notifications Dialog
5.3 Project Manager
Figure 45 Project Manager The SquidNet GUI uses a project based interface for all job submissions. Each submitted job from the GUI can be saved in a project folder where it can later be re-edited and resubmitted to the render farm job queue. Some of the advantages of using project folders include: Consolidation of jobs and transactions related to a single customer. Change processing parameters and re-submit the same job back to the render queue. Tracking of render scenes by project or customer. Access to the same project folder in mixed operating system render farms (thru use of the $XPATH() path translation macro) The Project Manager dialog (Project->Project Manager…) provides the following options: Select: Open selected project and close project manager window. Apply: Open selected project but keep project manager window open. New Project: Create a new project folder. Import: Add a pre-existing project folder. Refresh: Update project list.
74
SquidNet Network Render Manager
Figure 46 New Project dialog To create a new project folder: Press the "New Project" button (or right-click "New Project") to open the "Create new project folder" dialog. Enter the name of the new project folder. By default, the folder is saved in the "projects-dir" folder in the SquidNet configuration path. Press the OK button. To open the new project, just double-click on the project name that was just created. If using the same project folder between different operating systems, edit and project and add the $XPATH() macro. Example: $XPATH(MAYA-PROJECTS)\SquidNetProfiles\MayaProjects.
5.4 Application Path Manager
Figure 47 Application Path Manager Use the Application Path Manager (Admin->Path Configuration->Application Path Manager ...) to let SquidNet know where your render applications are installed. For SquidNet to process render farm requests you must create at least one application profile for each application that you’ll be submitting jobs to.
Figure 7 After Effects Application Profile (example) Each application profile contains the following information:
75
SquidNet Network Render Manager
Profile name: A unique name assigned to each profile (For example: MAYA-2012, 3DSMAX-2013 and MODO-701). Template type: The application template that the application profile is for. (For example: After Effects, Maya, etc…) Rendering application: The name of the template’s rendering application. Application Path: The absolute path to the rendering application (not including application name). Render node: The node or nodes where the application path can be found. This field can have the following values: Render Node Description Application path applies to all nodes. Use this setting if SquidNet can find the application in the same folder on all render farm nodes. Application path applies to all Windows nodes. Application path applies to all Mac OS nodes. Application path applies to all Linux nodes. Application path applies to the specified node name.
In most cases, each profile will have a single entry. For example, on an all-Windows render farm that has After Effects installed on each render node, the application profile will have the following properties:
Application Profile Name: AFTER-EFFECTS-CS6 Entry #1: o Render node: o Render Program: aerender.exe o Application Path: C:\Program Files\Adobe\Adobe After Effects CS6\Support Files
If you have a situation where, for example, After Effects is installed in the same path on all nodes except for node RENDERNODE14 where AE is installed on Drive E: then the application profile would have 2 entries:
Application Profile Name: AFTER-EFFECTS-CS6 Entry #1: o Render node: o Render Program: aerender.exe o Application Path: C:\Program Files\Adobe\Adobe After Effects CS6\Support Files Entry #2: o Render node: RENDERNODE14 o Render Program: aerender.exe o Application Path: E:\Program Files\Adobe\Adobe After Effects CS6\Support Files
If you also have a situation where you have AE installed on a Mac OS node, then the application profile will have 3 entries as follows:
Application Profile Name: AFTER-EFFECTS-CS6 Entry #1: o Render node: o Render Program: aerender.exe o Application Path: C:\Program Files\Adobe\Adobe After Effects CS6\Support Files Entry #2: o Render node: RENDERNODE14 o Render Program: aerender.exe o Application Path: E:\Program Files\Adobe\Adobe After Effects CS6\Support Files Entry #3: o Render node: MACOSNODE.LOCAL o Render Program: aerender o Application Path: /Applications/Adobe/Adobe After Effects CS6/Support Files
For the example above, you’ll need to reference the application profile in the job request by enclosing the profile name inside the $APP() macro (for example: $APP(AFTER-EFFECTS-CS6)). When SquidNet distributes the job to the render farm nodes, it’ll contain a copy of the application profile. When each render node receives the job request, it’ll attempt to find the best possible match.
5.4.1 Creating an application profile To create an application profile, press the “Create new path” button and do the following: 1. 2. 3.
Select the application type. Select the rendering program name Select the render node
76
SquidNet Network Render Manager
4. 5. 6.
Enter the absolute application path where the rendering program can be found (excluding rendering application name). Validate the path to make sure it exists. Press the OK button to register the profile.
Figure 7 After Effects Application Profile
5.5 Application Path Scanner
Figure 48 Application Path Scanner The figure above shows the Application Path Scanner (Admin->Path Configuration->Application Path Scanner ...). Use it to find applications installed on your render farm. When the scanner finds a particular application, you can either create a new application profile or add it to an existing profile. To use the scanner: 1. 2. 3. 4.
Select one or more applications to locate. Select the paths that you’d like to search. Either accept the defaults or supply additional paths. Press the “Scan Render Farm” button and wait for the scan to complete. Note: Depending on the size of your render farm, scanning can take a very long time. On completion, select one or more found applications, right-click to open popup menu then select to create a new application profile or add to an existing one.
5.6 Path Translation Manager
77
SquidNet Network Render Manager
Figure 49 Path Translation Manager Use the Path Translation Manager (Admin->Path Configuration->Path Translation Manager...) to create "mappings" between similar physical paths on different operating systems. The figure above shows the Path Translation Manager. As you can see, there's already a few path translation profiles that show the logical paths on WINDOWS, LINUX and OS X computers and point to the same physical folder on the network. For example:
Translation profile MAYA_PROJECTS maps the following paths on each machine to the same network folder: o WINDOWS: \\raid-server00\volume_1\SquidNet\MayaProjects o LINUX: /mnt/raid/MayaProjects o Mac OS: /Volumes/Volume_1/SquidNet/MayaProjects
In a mixed operating system environment, use path translation to allow SquidNet to properly map job requests between operating system paths. In your job request, just use the $XPATH() macro to refer to a specific translation profile (example: $XPATH(MAYA_PROJECTS), $XPATH(LIGHTWAVE_PROJECTS), etc...)
Figure 50 Maya job using application ($APP) and path ($XPATH) translation
78
SquidNet Network Render Manager
The figure above shows a cross-platform job using the $APP() and $XPATH() macros. To allow SquidNet to seamlessly process mixed-operating-system jobs requests, $APP() and $XPATH() macros must be used. In the example shown, the macros translate as follows:
$APP(MAYA-2011) maps to the following applications (as configured in Application Path Manager): o Mac OS Maya command line renderer application (Render) installed in /Applications/Autodesk/maya2011/Maya.app/Contents/MacOS folder on a specific Mac OS based node. o WINDOWS command line renderer application (render.exe) installed in C:\Program Files\Autodesk\Maya2011\bin folder on a specific WINDOWS based computer. $XPATH(SQUIDDIR) maps to the follow paths (all of which point to the same location on the network): o \\raid-server00\volume_1\SquidNet on a WINDOWS-based machine. o /mnt/raid/SquidNet on a LINUX-based machine. o /Volumes/Volume_1/SquidNet on an OS X based machine
Menu Item Add New Path Duplicate Path Edit Path Remove Path
Description Create new translation profile Duplicate an existing translation profile Edit and existing translation profile Remove translation profile
Table 10 Path Translation Manager Menu Options
5.7 Render Pool Manager
Figure 51 Render Pool Manager
79
SquidNet Network Render Manager
Figure 52 Render Pools The Render Pool Manager is used to combine slave nodes into specific groups of render nodes that perform specific actions.
Menu Item
Sub-menu item
Description
Table 11 Render Pool Manager Menu Options
80
SquidNet Network Render Manager
5.8 Image Sequence Viewer
Figure 53 Image Sequence Viewer
5.9 Window Views These windows are child windows of the main user interface and can be arranged within the main client area. Window view positions are maintained between UI restarts.
5.9.1 Project folder
Figure 54 Project View
81
SquidNet Network Render Manager
Menu Item
Sub-menu item
Description
Table 12 Project Folder Menu Options
Figure 55 Job Submission Form
5.9.2 Job Queue
Figure 56 Job Queue View
82
SquidNet Network Render Manager
Menu Item Job Request Job Slices Priority
Job Log
Sub-menu item Increase Decrease Change -
Description Open job request form. Open slices associated with job. Increase job priority in render queue. Decrease job priority in render queue. Change job priority in render queue. Open job’s activity log.
Table 13 Job Queue Menu Options
5.9.3 Job Slice Queue
Figure 57 Job Slice Queue View Menu Item
Sub-menu item
Description
Table 14 Job Slice Queue Menu Options
83
SquidNet Network Render Manager
5.9.4 Render Farm
Figure 58 Render Farm View Menu Item
Sub-menu item
Description
Table 15 Render Farm Menu Options
5.9.5 Network Work Queue
Figure 59 Network Work Queue View Menu Item
Sub-menu item
Description
Table 16 Network Work Queue Menu Options
84
SquidNet Network Render Manager
5.9.6 Image Preview
Figure 60 Image Preview View
5.9.7 Render Timeline
Figure 61 Render Timeline View
85
SquidNet Network Render Manager
5.9.8 CPU Usage
Figure 62 Render Farm CPU Usage View
5.9.9 Network Map
Figure 63 Render Farm Network Map View
86
SquidNet Network Render Manager
5.9.10 Job Stack
Figure 64 Job Stack View
87
SquidNet Network Render Manager
Chapter 6 CONFIGURATION
88
SquidNet Network Render Manager
6. CONFIGURATION To get SquidNet up and going, at a minimum, you’ll need to perform the following steps: 1. 2. 3. 4. 5.
Set the configuration path for all nodes from which jobs will be submitted from. Define which application templates you’ll be using. Create your initial project folder. Setup the SquidNet view layouts by opening (at a minimum) project folder view, job queue view, network work queue view and render farm view. Tell SquidNet where your applications “live” on your render farm by creating application profiles.
Other tasks that may need to be completed include: 1. 2.
Email/SMS setup Translation paths
6.1 Configuration path By default, the SquidNet UI stores all user configuration settings in the /settings folder. Configuration settings include:
Project folders and job profiles. Path translations. Application profiles. UI window layouts. Pool configurations. Email/SMS settings. etc...
It's recommended that the default configuration path be changed to an alternate location ----- preferably a network shared folder. If users will be submitting jobs from multiple client workstations, the same path should be set in the UI on each client node.
Figure 65 Configuration Path
89
SquidNet Network Render Manager
6.2 Application Templates SquidNet supports 3D applications thru application templates. Before you can submit jobs, you must determine which application templates you’ll be using. Use the Preferences dialog to declare which 3D application templates you'll be using.
6.3 License Registration Use the License Manager to register your SquidNet license(s). Open the License Manager (Help->License Manager...) and press the "Register New License" button. Cut/paste in your license.
6.4 Job Notifications SquidNet sends job notifications to users to let them know when specific workflow events occur. Use the notifications settings in the preferences dialog to determine which notifications types you’d like to receive. To view user notifications, select the “Notifications” button at the bottom of the UI. Note: button is hidden until at least one notification is received.
6.5 Setup default project Open the Project Manager (Project->Project Manager...) and create a new project by selecting the “New Project” folder. By default, projects are stored under the configuration path defined in the Preferences dialog. Once created, double-click on project to open project window.
6.6 Email setup
Figure 66 Email Configuration
6.7 Setting up master
90
SquidNet Network Render Manager
Chapter 7 PROJECT INTERFACE
91
SquidNet Network Render Manager User Interface Overview
7. PROJECT INTERFACE 7.1 Project Manager
Figure 67 Project Manager
7.2 Project Folder
Figure 68 Project Folder
92
SquidNet Network Render Manager User Interface Overview
Chapter 8
JOB SUBMISSION
93
SquidNet Network Render Manager
8. JOB SUBMISSION 8.1 Overview
Figure 69 Job Slice Efficiency
94
SquidNet Network Render Manager
8.2 Job Submission Form Processing Parameters
Application Parameters
Project settings
Application Template
Unique job identifier
Application specific fields Basic settings
Advanced settings
Figure 70 Job Submission Form
8.3 Overview The figure above shows the job submission form that’s used to submit job requests to the render farm. The form is separated into 2 sections:
Processing Parameters: Determine how SquidNet will process and manage job requests: o Template type: o Project Folder: o Scene/Shot name: o Job identifier: Auto Job Id Generation: Use Scene Name: Job Identifier: Job Description: Job Color Identifier: o Basic parameters: Max Slices: Slice Timeout(multiplier): Slice Timeout(mins): Slice Fails before Stop: Check Output Content: Frame Check: Use all cores: If checked, each jobslice will use all the available cores on the rendering computer. Start Suspended: Randomize Job Slices: Skip Existing Frames: Create Ouput Folder:
95
SquidNet Network Render Manager
Estimate Completion Time: o Priority of job in render queue. o Advanced options. o Completion action: Application Parameters: Determine how the rendering application will render the specified scene: o The selected template determines which fields are presented to user. o Red bounding boxes indicate mandatory fields. o Fields are template specific.
Processing parameters are as follows:
Template: The type of application that will process this request: o Select from the dropdown list. o Determine which templates are available in dropdown list by selecting from Preferences dialog. Project folder: The project that this job will belong to. Scene/shot name: Name of the shot that this scene belongs to. o By default, all scenes belong to the “Default” shot. o Use the “+” or “-“ buttons to the right of the shot name to add or remove shot names to the list. o All shot names are stored in the project folder.
96
SquidNet Network Render Manager
8.4 Processing Parameters 8.4.1 Template type 8.4.2 Project Folder 8.4.3 Scene/Shot name 8.4.4 Job identifier 8.4.4.1 Auto Job Id Generation 8.4.4.2 Use Scene Name 8.4.4.3 Job Identifier 8.4.4.4 Job Description 8.4.5 Basic parameters 8.4.5.1 Max Slices 8.4.5.2 Slice Timeout(multiplier) 8.4.5.3 Slice Timeout(mins) 8.4.5.4 Slice Fails before Stop 8.4.5.5 Check Output Content 8.4.5.6 Frame Check VALID filename formats 051013_00001.jpg thru 051013_00100.jpg MY_IMAGE_002_00001.exr thru MY_IMAGE_002_00999.exr TRUCK_0123_002_00001.exr thru TRUCK_0123_002_00999.exr
INVALID filename formats 05101300001.jpg thru 05101300100.jpg MY_IMAGE_00200001.exr thru MY_IMAGE_00200999.exr TRUCK_0123_00200001.exr thru TRUCK_0123_00200999.exr
97
SquidNet Network Render Manager
Figure 71 Valid and Invalid Output Image Formats
Features:
SquidNet will determine which node returned the “bad” frame(s) and will not send any more slices to that node. This is for the current job only.
8.4.5.7 Use all cores 8.4.5.8 Start Suspended 8.4.5.9 Randomize Job Slices 8.4.5.10 Skip Existing Frames 8.4.5.11 Create Ouput Folder 8.4.5.12 Estimate Completion Time 8.4.6 Priority 8.4.7 Advanced Options
Figure 72 Advanced Job Submission Options
98
SquidNet Network Render Manager
Advanced Option Copy/Move images on completion Dependendencies Email Notifications Environment variables Image Display Image File Permissions Memory Options Notes On Condition Profiles SMS Notifications Video generation for image sequences
Description Automatically copy or move rendered images (and sub-folders) from job’s output folder. Create job dependencies chains (allow specific jobs to complete in an ordered sequence). Send email notifications for job workflow events. Set environment variables for rendering application. Auto-display renderered images to local UI. Unix/Linux style file permission settings on output images. Settings for memory management on rendering nodes. Job request details, comments, etc… Initial job slice actions based on render conditions. Send email notifications for job workflow events. Auto-create H.264 videos from rendered images.
Figure 73 Advanced job request settings
8.4.7.1 Dependencies
Figure 74 Copy/Move Content
8.4.7.2 Email/SMS Notifications
99
SquidNet Network Render Manager
Figure 75 Email Notifications
Figure 76 SMS Notifications
8.4.7.3 Image Display
Figure 77 Image Display
100
SquidNet Network Render Manager
8.4.7.4 Memory Options
Figure 78 Memory Options
Memory usage: o Automatic requeueing of job slices that use more than the specified memory limit. The default is 97% which means that job slices will be automatically requeud if they consume more than 97% of available memory. It’s recommended that this setting be in place to prevent nodes from rendering job slices that may make the node unresponsive because of excesive memory paging. o Ability to remember the minimum memory used by a job slice and only submit slices to nodes that have equal or greater available RAM. o Ability to abort job slices if they were aborted multiple times because of insufficient memory. o The minimum amount of memory required to render job slices.
8.4.7.5 On Condition Profiles
Figure 79 On Condition Profiles
8.4.7.6 Video Generation
101
SquidNet Network Render Manager
Figure 80 Video Generation
8.4.8 Render Pools
Figure 81 Render Pool Select
102
SquidNet Network Render Manager
8.4.9 Include/Exclude
Figure 82 Processing Nodes
8.4.10 Tile Rendering
103
SquidNet Network Render Manager
Figure 83 Tile Render Setup
8.5 Application Parameters 8.5.1 Presets 8.5.2 Application 8.5.3 Frame List 8.5.4 Render Frames 8.5.5 Fields
104
SquidNet Network Render Manager
8.6 Submitting jobs 8.6.1 Scene preparation All texture references in scene files must be relative to the scene or located on a network share. Note: For the textures in the project scene files, make sure that they:
Have relative paths to project folder Have absolute paths to network share folders. DO NOT have textures paths that point to files stored on local disks. If you do, remote render nodes will not have access to these files. For example, do not have a reference to local disk file called “D:\my-textures\sky.png” in any scene file. If you do, the local SquidNet will render the scene just fine (because the file is local) but any remote nodes will fail to render the scene because they don’t have access to this texture.
8.6.2 GUI submission 8.6.2.1 Image Sequence Rendering
8.6.2.2 Tile Rendering
8.6.2.3 Video Generation
8.6.3 CLI submission 8.6.3.1 Image Sequence Rendering 8.6.3.2 Tile Render 8.6.3.3 Video Generation
105
SquidNet Network Render Manager
Chapter 9 TILE RENDERING
106
SquidNet Network Render Manager
9. TILE RENDERING Tile rendering is the rendering of a single image amongst 2 or more render nodes. With SquidNet Tile Rendering it’s possible to have multiple render nodes work on different portions of the same image and then have SquidNet “stitch” the images back together once completed. SquidNet Tile Rendering offers the following features:
Tile Render Image Viewer that shows progress of tile render jobs. Supported by any application that supports region rendering (Maya, modo, 3DSMAX, Terragen, and LightWave). Ability to specify multiple frames (single frames are the default) User-selectable rows and columns. User-selectable tiles within image. Select custom or pre-defined image resolutions. Support for most image formats (bmp, png, jpeg, exr, hdr, tif, gif, etc…) Auto-stitching of final image. Submission from GUI or CLI.
Tile rendering guidelines:
Render heavy frames that may take hours to render if using a single render node. Improve network efficiency by rendering larger images (3000x2000) or greater. Do not render low-res images because the overhead involved in tracking each tile.
107
SquidNet Network Render Manager
Chapter 10 FARM MANAGEMENT
108
SquidNet Network Render Manager
10. FARM MANAGEMENT
109
SquidNet Network Render Manager
Chapter 11 TUTORIAL
CROSS-PLATFORM
110
SquidNet Network Render Manager
11. CROSS-PLATFORM TUTORIAL
11.1 Overview
Figure 84 Cross-platform Render Farm Network SquidNet provides cross-platform support meaning that you can configure a render farm that's running under different operating systems. Each computer on the farm can run any version of Windows, Mac OS or Linux and each node can be configured to be a master, client or slave render node. This section will guide you through the process of configuring SquidNet to run in a cross-platform environment. It’ll show you how to do the following:
Configure After Effects, Maya and Modo on all rendering nodes. Setup network shares for scene content and rendered image storage. Copy sample AE, Maya and Modo scene files from the SquidNet installation folder onto network share folders. Create SquidNet SSA account on all Windows nodes. Install SquidNet on all nodes. Configure master, client and slave nodes from GUI and CLI.
111
SquidNet Network Render Manager
Setup default configuration path. Setup application profiles that tell SquidNet where rendering applications are located. Setup translation profiles that map network share paths for each platform. Submit rendering jobs from GUI and CLI interfaces. Monitor job progress using tools from GUI and CLI interfaces. Create H.264/MPEG4 video of animation sequences from GUI and CLI interfaces. Submit tile rendering job.
11.2 Assumptions For this tutorial, the following assumptions are made:
Project folders will be stored on network shares: o Windows: \\raid-server00\volume_1\SquidNet o Mac OS: /Volumes/Volume_1/SquidNet o Linux: /mnt/SquidNet Render farm nodes will be configured as follows: o MASTER: SQUIDNET-WIN-MASTER Running on a Windows-based computer. The master will not be processing jobs. o CLIENTS: SQUIDNET-WIN-CLIENT Windows client will be configured to never process jobs. SQUIDNET-OSX-CLIENT Mac OS client will be configured to process jobs when screensaver is active. o SLAVES SQUIDNET-WIN-SLAVE[01-05] SQUIDNET-LINUX-SLAVE[01-03] Running on latest Ubuntu and CentOS Linux distributions Rendered images will be saved to: o Windows: \\raid-server00\volume_1\SquidNet\Tutorial-Images o Mac OS: /Volumes/Volume_1/SquidNet/Tutorial-Images o Linux: /mnt/SquidNet/Tutorial-Images The rendering applications will be installed on the following platforms: o After Effects: Windows and Mac OS. o Autodesk Maya: Windows and Mac OS. o Luxology Modo: Windows, Mac OS and Linux.
11.3 Installing 3rd Party Application(s) SquidNet requires that all 3rd party rendering applications (Maya, After Effects, LightWave, modo, etc…) be installed on every render node. This is necessary because the local SquidNet service must have access to the application’s rendering program. The rendering program is the command line tool that SquidNet calls to process scene renders. Most 3rd party applications provide a separate renderer program that resides in the same folder as the main application GUI. In most cases, the command line renderer program is different than the main application GUI. The table below shows the renderer programs supplied by various 3D and compositing applications: 3D/Compositing Application After Effects Autodesk 3DSMAX Autodesk Maya Blender3D CINEMA 4D LightWave Luxology Modo Nuke Terragen V-Ray Standalone
WINDOWS RENDERER APP aerender.exe 3dsmaxcmd.exe render.exe blender.exe "CINEMA 4D.exe" lwsn.exe modo_cl.exe nuke.exe tgdcli.exe vray.exe
Mac OS RENDERER APP aerender render blender “CINEMA 4D” lwsn modo_cl "Terragen 2" vray
112
Linux RENDERER APP Render blender modo_cl vray
SquidNet Network Render Manager
XSI/SoftImage
xsibatch.bat
-
-
Table 17 3rd Party Application Rendering Programs (not all applications are listed) The renderers listed above are what SquidNet will use to render your job requests. When you get to the application path manager step, you'll need to tell SquidNet where these applications are installed on your render farm by specifying the paths in your application profile definitions.
11.4 Setting up a network folder Make sure that each operating systems has read/write access to following network shares:
Windows: \\raid-server00\volume_1\SquidNet Mac OS: /Volumes/SquidNet Linux: /mnt/SquidNet
Each partition should point to the same physical folder on the network.
11.5 Setup tutorial scenes In SquidNet's installation folder, copy the "Tutorials" folder (including sub-folders) to the network share folder. When the copy is complete, you should have the following folders:
Windows: \\raid-server00\volume_1\SquidNet\Tutorials Mac OS: /Volumes/SquidNet/Tutorials Linux: /mnt/SquidNet/Tutorials
11.6 Setting up SSA account On Windows render nodes, create an admin account (SSA) named “SquidNet” with a password of your choice. Once the account is created, log out of your current Windows session and log in again using the new SSA account. Make sure you can access any network folders on the network share folders. Try creating and removing some files on the network share while logged in under the SSA account. If you can access the network share without issue, you should be good to go !
11.7 Installing SquidNet Using the instructions that are available on the SquidNet website, install SquidNet on each render farm node.
SquidNet Installation Guide for Windows. (open) SquidNet Installation Guide for Linux. (open) SquidNet Installation Guide for Mac OS. (open)
On Windows computers, make sure you enter the newly created SSA account when prompted.
113
SquidNet Network Render Manager
11.8 Configure render farm nodes
Table 18 Initial Render Farm Configuration At this point you should have SquidNet installed on all you render nodes. Now, on the master node, start the main SquidNet GUI and open up the Render Farm Nodes view from the main menu tool bar (Network->Render Farm ...). Note: Node configuration must be done on the master computer. Make sure you have the columns displayed as shown above. If not, enable any missing columns by right-clicking on the Render Farm view and selecting "Select Columns..." from popup menu. The next steps will be to configure each render node. Do the following in order: 1. 2.
3.
4.
Right-click on SQUIDNET-WIN-MASTER node and select Configuration->Set as Master... menu option. Right-click on SQUIDNET-WIN-CLIENT node and select Configuration->Set as Client... menu option. a. Since we don't want our client node to render jobs, right-click on SQUIDNET-WIN-CLIENT node and select Configuration->Configure Client... menu option. Choose "Never" from dropdown menu. Right-click on SQUIDNET-OSX-CLIENT node and select Configuration->Set as Client... menu option. a. Since we don't want our client node to render jobs, right-click on SQUIDNET-OSX-CLIENT node and select Configuration->Configure Client... menu option. Choose "Screensaver" from dropdown menu. Select all slave nodes, right-click and select Configuration->Set as Slave... menu option.
The GUI commands above could have been done using the following CLI commands: CLI Command
Description
sqn --resetFarm sqn --masterSet SQUIDNET-WIN-MASTER sqn --nodeTypeSet client --node SQUIDNET-WIN-CLIENT sqn --clientModeSet never --node SQUIDNET-WIN-CLIENT sqn --nodeTypeSet client --node SQUIDNET-OSX-CLIENT sqn --clientModeSet screensaver --node SQUIDNET-OSXCLIENT sqn --nodeTypeSet slave --node SQUIDNET-WIN-SLAVE01 sqn --nodeTypeSet slave --node SQUIDNET-WIN-SLAVE02 sqn --nodeTypeSet slave --node SQUIDNET-WIN-SLAVE03 sqn --nodeTypeSet slave --node SQUIDNET-WIN-SLAVE04 sqn --nodeTypeSet slave --node SQUIDNET-WIN-SLAVE05 sqn --nodeTypeSet slave --node SQUIDNET-LINUX-SLAVE01 sqn --nodeTypeSet slave --node SQUIDNET-LINUX-SLAVE02 sqn --nodeTypeSet slave --node SQUIDNET-LINUX-SLAVE03
Resets ALL render farm nodes to PEERs Sets node SQUIDNET-WIN-MASTER to be the SquidNet master Sets node SQUIDNET-WIN-CLIENT to be a SquidNet client Prevents node SQUIDNET-WIN-CLIENT from processing jobs Sets node SQUIDNET-OSX-CLIENTto be a SquidNet client Allows node SQUIDNET-OSX-CLIENT to render jobs when screen saver is running Sets node SQUIDNET-WIN-SLAVE01 to be a SquidNet slave Sets node SQUIDNET-WIN-SLAVE02 to be a SquidNet slave Sets node SQUIDNET-WIN-SLAVE03 to be a SquidNet slave Sets node SQUIDNET-WIN-SLAVE04 to be a SquidNet slave Sets node SQUIDNET-WIN-SLAVE05 to be a SquidNet slave Sets node SQUIDNET-LINUX-SLAVE01 to be a SquidNet slave Sets node SQUIDNET-LINUX-SLAVE02 to be a SquidNet slave Sets node SQUIDNET-LINUX-SLAVE03 to be a SquidNet slave
114
SquidNet Network Render Manager
Table 19 Configured Render Nodes
11.9 Register license If you have a SquidNet license, you can register it as follows:
Open GUI on master node. Open License Manager dialog (Help->License Manager...) Press "Register New License" button and cut/paste in your SquidNet license(s).
11.10 Set up default configuration path
Figure 4 Configuration Path The configuration path is where the SquidNet UI will save all your GUI settings (jobs, presets, pools, etc...). To avoid having to duplicate all GUI settings, make sure you set the configuration path to a network shared folder. On the master and on every client node, open the Preferences dialog and set the Configuration Path to (create the “SquidNetConfig” folder if it doesn’t already exist):
SQUIDNET-WIN-MASTER and SQUIDNET-WIN-CLIENT: o \\RAID-SERVER00\volume_1\Squidnet\SquidNetConfig SQUIDNET-OSX-CLIENT o /Volumes/SquidNet/SquidNetConfig Not needed on slave nodes
Press the "Apply" button to set the path.
115
SquidNet Network Render Manager
11.11 Set up job templates
Figure 5 Job Templates With the SquidNet preferences dialog still open, select the "Job Submission" option on the left side of the dialog and add the After Effects, FFMPEG, Maya and Modo templates to the right side list. Do this on both the master and client nodes. This setting determines which job templates are available in your project window and which applications you’ll be able to submit jobs to.
11.12 Set up application profiles
Figure 6 Application Path Manager We'll need to setup profiles to tell SquidNet where to find our applications so open up the Application Path Manager and register our AE, Maya and Modo render applications. Do this on either the master or one of the client nodes. If the configuration path is set up correctly, any profiles created here will be available for all nodes to use.
Open Application Path Manager (Admin->Path Configuration->Application Path Manager ...) For After Effects, create an application profile with the following settings: o Profile Name SQUIDNET-TUT-AFX
Application aerender.exe
Render Node
SQUIDNET-TUT-AFX
aerender
Renderer Path C:\Program Files\Adobe\Adobe After Effects CS6\Support Files /Applications//Adobe After Effects CS6
o For Maya, create an application profile with the following settings: o Profile Name SQUIDNET-TUT-MAYA
Application render.exe
Render Node
116
Renderer Path C:\Program Files\Autodesk\Maya2014\bin
SquidNet Network Render Manager
SQUIDNET-TUT-MAYA
Profile Name SQUIDNET-TUT-MODO SQUIDNET-TUT-MODO SQUIDNET-TUT-MODO
Render
o For Modo, create an application profile with the following settings: o Application modo_cl.exe modo_cl modo_cl
Render Node
Renderer Path C:\Program Files\Luxology\modo\701_sp1 /Applications/modo.app/Contents/MacOS/ /downloads/modo/
o The paths above are assumed so if your installation paths are different, update accordingly. Note: SquidNet will not submit slices to nodes that are missing render application. Make sure to "validate" each profile. Press "OK" button to save profile(s)
11.13 Translation Profiles Translation paths allow SquidNet to properly "map" folder paths between different operating systems. For example, consider the following paths: o o o
Windows: \\raid-server00\volume_1\SquidNet Mac OS: /Volumes/Volume_1/SquidNet Linux: /mnt/SquidNet
If you submit a job with the /Volumes/Volume_1/SquidNet path to a Windows render node it'll fail because Windows won't know to handle it because it only understands folder paths in the \\raid-server00\volume_1\SquidNet format. Likewise, passing a path like \\raid-server00\volume_1\SquidNet to a Linux render won't work for the same reason. So to get around this, SquidNet uses translation profiles to map folder paths in job requests so that each render node can recognize its perspective path format. To use translation profiles just include the profile name in the $XPATH () macro. To create a translation profile, open the Path Translation Path Manager (Admin->Path Configuration->Path Translation Manager...) and press the "Create new translation" button. Create a profile called "SQUIDNET-TUT-PATH".
Figure 85 Path Translation Create The SQUIDNET-TUT-PATH profile simply states that all 3 paths point to the same physical network folder.
11.14 Create project folder All SquidNet render jobs are stored in project folders. So before we can submit our render jobs, we'll need to create a project folder:
117
SquidNet Network Render Manager
Open Project Manager dialog (Project->Project Manager...) Press the "New Project" button and create a project named "SQUIDNET-CROSS-TUTORIAL" Double-click on project "SQUIDNET-CROSS-TUTORIAL" in the Project Manager dialog to open a project view for project "SQUIDNET-TUTORIAL"
Figure 10 New Project "SQUIDNET-TUTORIAL"
11.15 Submit job requests
Figure 11 Project "SQUIDNET-TUTORIAL"
11.15.1 After Effects Job We're now ready to formally submit jobs to the render farm. With the view for project "SQUIDNET-TUTORIAL" open in the GUI client area, press the "New Job" button and select the After Effects template from the top-left of the job submission form.
118
SquidNet Network Render Manager
Figure 12 After Effects Job Submission On the job submission form, enter or change the following fields: 5.
6.
7. 8.
On left-side of the form: a. Set the "Job ID" to AFX-JOB#1 b. Select "Create output folder" On right-side of the form, set the following fields: a. Application: $APP(SQUIDNET-TUT-AFX) b. Project File: $XPATH(SQUIDNET-TUT-PATH)/Tutorials/AfterEffects/fire.aep c. Start Frame: 1 d. Ending Frame: 400 e. Frame Per Slice: 1 f. Output Folder: $XPATH(SQUIDNET-TUT-PATH)/Tutorial-Images/AE g. Output Filename: fire[####].jpg h. OSX username (mac rendering): i. Under "Other settings" panel: i. Render Queue Index: 1 Press "Submit" button. When asked to save job, say "yes"
Equivalent CLI Command sqn --submit --template AE --appPath $APP(SQUIDNET-TUT-AFX) --jobId AFX-JOB#2 --startFrame 1 --endFrame 400 --priority 12 --framePerSlice 1 --createOutputDir --sceneFile //RAID-SERVER00/Volume_1/SquidNet/Tutorials/AfterEffects/fire.aep -outputDir //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/AE --videoFilename "fire[####].jpg" --renderIndex 1 -useAllCores
11.15.2 Maya Job Press the "New Job" button and select the Maya template from the top-left of the job submission form.
119
SquidNet Network Render Manager
Figure 13 Maya Job Submission On the job submission form, enter or change the following fields: 5.
6.
7. 8.
On left-side of the form: a. Set "Job ID" to MAYA-JOB#1 b. Select "Use All Available Cores" c. Select "Create output folder" On right-side of the form, set the following fields: a. Application: $APP(SQUIDNET-TUT-MAYA) b. Project Dir: //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Maya c. Scene: //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Maya/fluidParticleFalls.ma d. Working Directory: //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Maya e. Start Frame: 1 f. Ending Frame: 450 g. Frame Per Slice: 20 h. Under "General Purpose" panel: i. Renderer to Use: Software ii. Output Image Path: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Maya iii. Format: name#.ext iv. Type: png Press "Submit" button. When asked to save job, say "yes"
Equivalent CLI Command sqn --submit --template maya --jobId MAYA-JOB#2 --useRenderer Software --appPath $APP(SQUIDNET-TUT-MAYA) --sceneFile //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Maya/fluidParticleFalls.ma --projectFolder //RAIDSERVER00/Volume_1/SquidNet/Tutorials/Maya --workDir //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Maya --startFrame 1 --endFrame 450 --framePerSlice 20 --imageFilenameFormat name#.ext --priority 12 --outputDir //RAIDSERVER00/Volume_1/SquidNet/Tutorial-Images/Maya --imagePrefix fluidparticles_ --imageFormat png --useAllCores -createOutputDir
120
SquidNet Network Render Manager
11.15.3 Modo Job Press the "New Job" button and select the Modo template from the top-left of the job submission form.
Figure 14 Modo Job Submission On the job submission form, enter or change the following fields: 5.
6.
7. 8.
On left-side of the form: a. Set "Job ID" to MODO-JOB#1 b. Select "Create output folder" On right-side of the form, set the following fields: a. Application: $APP(SQUIDNET-TUT-MODO) b. Scene File: //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Modo/racecar.lxo c. Important: Press "Scan Modo scene" button. d. Start Frame: 1 e. Ending Frame: 300 f. Frame Per Slice: 10 g. Output Image Path: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo h. Image Prefix: racecar_ i. Output Image Format: png j. Render Camera: OVERHEAD Press "Submit" button. When asked to save job, say "yes"
Equivalent CLI Command sqn --submit --jobId MODO-JOB#2 --template modo --appPath $APP(SQUIDNET-TUT-MODO) --sceneFile //RAIDSERVER00/Volume_1/SquidNet/Tutorials/Modo/racecar.lxo --startFrame 1 --endFrame 300 --framePerSlice 10 --priority 12 -outputDir //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo --imagePrefix racecar_ --imageFormat png -useAllCores --createOutputDir --camera OVERHEAD
121
SquidNet Network Render Manager
11.16 Monitoring render farm At this point, you should have 3 jobs submitted to the render farm job queue. In this section, we'll show you some basics on how to monitor and manage these jobs from both the GUI and CLI interfaces. Start the SquidNet GUI and open the following views:
Image Viewer: Tools->Image Preview.. Project view for project "SQUIDNET-TUTORIAL": Project->Project Manager ... Render Farm Nodes: Network->Render Farm... Job Queue: Tools->Job Queue... Network Work Queue: Network->Network Work Queue...
Arrange and resize views in the GUI client area so that all windows are visible. Alternatively, you can select Window->Tile Windows... to auto-arrange windows for you.
11.16.1 GUI monitoring
Figure 15 SquidNet User Interface There's over a dozen window views that you can select from but for the sake of this tutorial we're going to focus on these:
Project View: Image Preview: Job Queue: Render Farm Nodes: Network Work Queue:
122
SquidNet Network Render Manager
11.16.1.1 Project View
The Project view will contain the jobs that we've already submitted. Once saved, you can always open up any jobs in the project, edit the processing parameters and resubmit to the farm.
11.16.1.2 Job Queue View
Equivalent CLI Command sqn --jobQueueShow Job Id Position State Progress Frames Slices Avg Slice Time GHZ-Hours Est.Proc.Time Est.Comp.Time --------------------------------------------------------------------------------------------------------------------------------------------------AFX-JOB#2 1 Processing 42% 1-400 400 25s 26.428774 01h.54m.10s 01h.54m.10s MAYA-JOB#2 2 Processing 21% 1-450 23 04m.50s 36.344572 02h.40m.33s 04h.34m.43s
The Job Queue view lists all the jobs in the render farm queue. Press the arrow button in the upper right corner of the view to open the job slice side panel. Double-click any job in the job queue to list the associated job slices.
123
SquidNet Network Render Manager
11.16.1.3 Job Slices
Equivalent CLI Command sqn --jobId MODO-JOB#1 --jobSliceShow # Frames State RENDERNODE TIME GHZ-hrs PRI Memory Used -------------------------------------------------------------------------------------------------1 0-9 Completed RENDERNODE14 01m 28s 0.665084 12 446980 KB 2 10-19 Completed RENDERNODE13 01m 39s 0.748220 12 453312 KB 3 20-29 Completed RENDERNODE11 02m 46s 1.033258 12 446124 KB 4 30-39 Completed RENDERNODE10 03m 21s 1.251113 12 444532 KB 5 40-49 Completed RENDERNODE12 03m 40s 1.369378 12 430544 KB 6 50-59 Completed RENDERNODE14 02m 18s 1.042973 12 448512 KB 7 60-69 Completed RENDERNODE13 02m 33s 1.156340 12 453156 KB 8 70-79 Completed RENDERNODE11 03m 49s 1.425398 12 491912 KB 9 80-89 Processing RENDERNODE10 1.313358 12 0 KB 10 90-99 Processing RENDERNODE12 1.232440 12 0 KB 11 100-109 Completed RENDERNODE14 02m 18s 1.042973 12 520700 KB 12 110-119 Completed RENDERNODE13 02m 15s 1.020300 12 517204 KB 13 120-129 Processing RENDERNODE14 0.347658 12 0 KB 14 130-139 Processing RENDERNODE13 0.211618 12 0 KB 15 140-149 Processing RENDERNODE11 0.155611 12 0 KB 16 150-159 Queued 12 0 KB 17 160-169 Queued 12 0 KB 18 170-179 Queued 12 0 KB 19 180-189 Queued 12 0 KB 20 190-199 Queued 12 0 KB 21 200-209 Queued 12 0 KB 22 210-219 Queued 12 0 KB 23 220-229 Queued 12 0 KB 24 230-239 Queued 12 0 KB 25 240-249 Queued 12 0 KB 26 250-259 Queued 12 0 KB 27 260-269 Queued 12 0 KB 28 270-279 Queued 12 0 KB 29 280-289 Queued 12 0 KB 30 290-299 Queued 12 0 KB 31 300-300 Queued 12 0 KB
In the job queue view, right-click on one of the rendering jobs and select the "Job Slices..." option from popup menu. This will bring up the job slice view which shows the status of each job slice.
124
SquidNet Network Render Manager
11.16.1.4 Application log
With the Job Slice window still open, right-click on any of the completed jobs slices and select the "Application log..." option from the popup menu. This will open up the application's rendering log.
11.16.1.5 Render Farm View
Equivalent CLI Command sqn --renderFarmShow Node IP Type O/S State Version Cores CPU Usage Active Processing GHZ/hrs Memory Free Uptime -----------------------------------------------------------------------------------------------------------------------------------------------------------------------THOR 10.0.0.28 MASTER WIN8x64 Idle 2.66P102 12 10 2 0 38.424000 67050720 63360768 11h 55m 56 RENDERNODE10 10.0.0.90 SLAVE VISTA64 Busy 2.66P102 8 95 0 1 22.344000 8345796 5445272 01h 50m 39 RENDERNODE12 10.0.0.92 SLAVE VISTA64 Busy 2.66P102 8 7 0 1 22.344000 8345796 6975236 01h 50m 33 RENDERNODE11 10.0.0.91 SLAVE VISTA64 Busy 2.66P102 8 93 0 1 22.344000 8345796 5361320 01h 50m 21 RENDERNODE13 10.0.0.93 SLAVE VISTA64 Busy 2.66P102 8 64 0 1 27.200000 8055948 6032168 01h 50m 50 RENDERNODE02 10.0.0.82 CLIENT VISTA64 Idle 2.66P102 8 3 0 0 27.200000 8055948 6892580 01h 51m 00 RENDERNODE14 10.0.0.94 SLAVE VISTA64 Busy 2.66P102 8 60 0 1 27.200000 8055948 6208360 01h 50m 58
The Render Farm view shows that status of all render farm nodes. If your submitted jobs are still running, you should see the state of all slave nodes set to "Busy"
125
SquidNet Network Render Manager
11.16.1.6 Farm Work Queue View
Equivalent CLI Command sqn --netQueueShow Node Job QueTime Source Template Jobslice ----------------------------------------------------------------------------------------------------------------------------RENDERNODE10 MAYA-JOB#2 29m 56s THOR Autodesk Maya 8 RENDERNODE11 MAYA-JOB#2 30m 38s THOR Autodesk Maya 7 RENDERNODE12 AFX-JOB#2 21s THOR Adobe After Effects 186 RENDERNODE13 MAYA-JOB#2 09m 04s THOR Autodesk Maya 9 RENDERNODE14 MAYA-JOB#2 01m 17s THOR Autodesk Maya 10
The Farm Work Queue view shows a list of all active render nodes and the jobs that they're currently working on.
11.16.1.7 Image Preview View
The Image Preview view displays the output images from the output folder of any selected job. Click on any job in the project or job queue views and notice how the Image Preview window starts scanning the output folder and displays the selected jobs rendered images. If more than a few images have been rendered, press the "Play" button to play back the image sequence in real-time.
126
SquidNet Network Render Manager
11.17 Video from image sequences This section will show you how to create video files from your image sequences.
11.17.1 Playback application In order to auto-playback rendered videos, set the path to your favorite video player app in the preferences dialog (Admin>Preferences...).
Figure 16 Video Player Path Setting For this tutorial we'll be using the free Video Lan Player (VLC) player. In this case, path is set to "C:/Program Files (x86)/VideoLAN/VLC/vlc.exe"
11.17.2 After Effects Video Press the "New Job" button and select the FFMPEG template from the top-left of the job submission form.
Figure 17 After Effects Video Creation
127
SquidNet Network Render Manager
5.
6.
7. 8.
On left-side of the form: a. Set "Job ID" to AFX-VIDEO#1 b. Select "Create output folder" On right-side of the form, set the following fields: a. Frame rate: 24 fps b. Image Sequence Folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/AE c. Video output folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Videos d. Select "Play Video" button. Press "Submit" button. When asked to save job, say "yes"
Equivalent CLI Command sqn --submit --jobId AFX-VIDEO#2 --template ffmpeg --appPath ffmpeg --inputDir //RAIDSERVER00/Volume_1/SquidNet/Tutorial-Images/AE --outputDir //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Videos --videoFrameRate "24 fps" --videoResolution "hd720(1280x720)" --useAllCores --videoPlay
11.17.3 Maya Video This time, instead of creating a job from a template, we’re going to create a new FFMPEG job an existing job profile. In the SQUIDNET-TUTORIAL project, open the AFX-VIDEO#1 job profile created in the previous step and change the following fields:
Figure 18 Maya Video Creation 5.
6.
On left-side of the form: a. Set "Job ID" to MAYA-VIDEO#1 b. Make sure "Create output folder" is checked. On right-side of the form, set the following fields: a. Frame rate: 24 fps b. Image Sequence Folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Maya
128
SquidNet Network Render Manager
7. 8.
c. Video output folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Videos d. Select "Play Video" button. Press "Submit" button. When asked to save job, say "yes"
Equivalent CLI Command sqn --submit --jobId MAYA-VIDEO#2 --template ffmpeg --appPath ffmpeg --inputDir //RAIDSERVER00/Volume_1/SquidNet/Tutorial-Images/Maya --outputDir //RAID-SERVER00/Volume_1/SquidNet/TutorialImages/Videos --videoFrameRate "24 fps" --videoResolution "hd720(1280x720)" --useAllCores --videoPlay
11.17.4 Modo Video In the SQUIDNET-TUTORIAL project, open the AFX-VIDEO#1 job profile created in the previous step and change the following fields:
Figure 19 Modo Video Creation 5.
6.
7. 8.
On left-side of the form: a. Set "Job ID" to MODO-VIDEO#1 b. Make sure "Create output folder" is checked. On right-side of the form, set the following fields: a. Frame rate: 24 fps b. Image Sequence Folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Maya c. Video output folder: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Videos d. Select "Play Video" button. Press "Submit" button. When asked to save job, say "yes"
Equivalent CLI Command sqn --submit --jobId MODO-VIDEO#2 --template ffmpeg --appPath ffmpeg --inputDir //RAID-
129
SquidNet Network Render Manager
SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo --outputDir //RAID-SERVER00/Volume_1/SquidNet/TutorialImages/Videos --videoFrameRate "24 fps" --videoResolution "hd720(1280x720)" --useAllCores --videoPlay
11.18 Modo Tile Rendering To render a single image across multiple render nodes, do the following:
Figure 20 Modo Tile Render Form
130
SquidNet Network Render Manager
Figure 21 Modo Tile Render Setup
Figure 22 Modo Tile Render Viewer 8. 9.
In the SQUIDNET-TUTORIAL project folder, open the MODO-JOB#1 job that was created earlier. On left-side of the form: a. Change "Job ID" to MODO-JOB-TILE-RENDER#1 b. Make sure "Create output folder" is checked.
131
SquidNet Network Render Manager
10. On right-side of the form, set the following fields: a. Application: $APP(SQUIDNET-TUT-MODO) b. Project File: //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Modo/racecar.lxo c. Start Frame: 1 d. Ending Frame: 1 e. Frame Per Slice: 1 f. Output Image Path: //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo-Tile-Render g. Image Prefix: racecar_ h. Output Image Format: png i. Render Camera: OVERHEAD 11. On left-side of the form: a. Press the "Tile Rendering" button in “Render Options” section. b. In "Tile Rendering Setup" dialog set the following values: i. Resolution Preset: HD 720 ii. Format: png iii. Move row slider to select 8 rows. iv. Move col slider to select 5 columns. v. Press "Select All" button to select all rows and columns. Alternatively, use the mouse to select specific cells in the grid. vi. Set "Enable" checkbox. vii. Press "Apply" button. 12. Press "Submit" button. 13. When asked to save job, say "yes" 14. In Job Queue view, right click on the newly submitted MODO-JOB-TILE-RENDER#1 job and select "Tile Render Viewer" to see the tile render operation in real-time. Equivalent CLI Command sqn --submit --sceneName MODO-JOB-TILE-RENDER#2 --template modo --tileRequest --tileRenderer "modo" --tileRows 8 -tileCols 5 --tileMode grid --tileRegions all --tileImagePrefix racecar_ --tileImageFmtIdx 1 --appPath $APP(SQUIDNET-TUTMODO) --sceneFile //RAID-SERVER00/Volume_1/SquidNet/Tutorials/Modo/racecar.lxo --startFrame 1 --endFrame 1 --priority 4 --outputDir //RAID-SERVER00/Volume_1/SquidNet/Tutorial-Images/Modo-Tile-Render --imagePrefix racecar_$(REG_MASK) -imageWidth 1280 --imageHeight 720 --imageFormat png --useAllCores --include
132
SquidNet Network Render Manager
Chapter 12
TRANSACTIONS
133
SquidNet Network Render Manager
Chapter 13
COMMAND LINE INTERFACE
134
SquidNet Network Render Manager
12. Command Line Interface SquidNet’s “sqn” command line interface mimics all operations and options available in the main graphical user interface. You can issue commands by appending specific options or by specifying options in a text file. Use "sqn --help" command to list all available options. All options are prefixed with “--“ (double-dash) characters.
13.1 Commonly used commands Management --masterSet --masterProcess --nodeTypeSet --clientModeSet --configPathSet
Description Sets the specified node as the render farm master. Enable or disables ability of master to process job requests Set render node type (master, client, slave or peer) Sets client node processing mode (none, background, etc…) Sets configuration path folder.
Table 20 Configuration Job submission --submit --pack --tileRequest --transaction --video
Description Send job request to render farm queue. Archive selected folder. Send tile render request to render farm queue. Send transaction request to render farm queue. Create video file from image sequence.
Table 21 Job Submission Management --jobQueueShow --renderFarmShow --status --lock/--unlock
Description Shows render farm job queue contents. Shows render farm nodes. Shows render farm status. Prevent nodes from processing job requests.
Table 22 Render Farm Management
13.2 Standard command line options The follow table lists standard command line options. Command appPathShow
Parameter
Description and example Displays application path profiles. Example: Sqn --appPathShow
appProfileTemplateShow
Displays application profiles associated with specified template. Example: List all application profiles for 3DSMAX template. sqn --appProfileTemplateShow 3dsmax Note: To get list of application templates, use this command: sqn --template all
aspectRatio
Sets the image or video aspect ratio. Example: Create MPEG4 videos with specified frame-rate, resolution and aspect-ratio. sqn --video S:\Images\flames --videoOutputDir S:\Archives\Videos --
135
SquidNet Network Render Manager
videoFrameRate 29 --videoResolution 800x600 --aspectRatio 4:3 --codec mpeg4 --noWait cancel
Cancel job or transaction request. Example: Cancelling a render job: sqn --jobId #1324 --cancel Example: Cancelling a transaction sqn --transactionId update-maya-install-base --cancel
cancelDependencies
When canceling a job, also cancel any dependent jobs. Example: Cancel job id maxJob#1 and also cancel any jobs that may be dependent on it. sqn --jobId maxJob#1 --cancel --cancelDependencies
clearDependencies
When canceling a job, prevent and child jobs from still being dependent on specified job. Example: Cancel job id maxJob#1 and clear any dependencies on it. sqn --jobId maxJob#1 --cancel --clearDependencies
clientModeSet
Sets the processing mode for SquidNet client nodes. Possible options are: Client Mode never loggedout background
Description Prevents node from rendering jobs. Only render jobs when all users are logged out. Render when user is logged in but render at a very low CPU processing priority. Only render jobs when screen saver is running. (Mac OS only)
screensaver
Example: Set all client nodes to render jobs at a low processing priority. sqn --clientModeSet background --node clients Example: Set client node RENDERNODE10 so that it only renders jobs when the screen saver is active (Mac OS only). sqn --clientModeSet screensaver --node RENDERNODE10 Example: Set client node RENDERNODE10 so that it can never render jobs. sqn --clientModeSet never --node RENDERNODE10 codec
Codec to use when creating video from image sequences. Example: Create MPEG4 videos with specified frame-rate, resolution and aspectratio. sqn --video S:\Images\flames --videoOutputDir S:\Archives\Videos -videoFrameRate 29 --videoResolution 800x600 --aspectRatio 4:3 --codec mpeg4 --noWait
column
Used on some CLI commands to only display specific columns. Example: Only display Job id, frames, slices and owner columns for job queue listing. sqn --jobQueueShow --column "Job Id",Frames,Slices,Owner
136
SquidNet Network Render Manager
columnShow
Display columns that could be displayed when using "--csv" option. Example: sqn --jobHistoryShow --columnShow Example: sqn --renderFarmShow --columnShow Example: sqn --jobQueueShow --columnShow
compress
Used with "--pack" to compress output file. Example: : Archive and compress S:\Images\0030-XSI-Fire-Ring folder to create E:\temp\0030-XSI-Fire-Ring.tar.gz archive file. sqn --pack S:\Images\0030-XSI-Fire-Ring --outputFile E:\temp\0030-XSI-FireRing.tar --compress
configPathSet
Sets the configuration path on specified nodes. The configuration path is where all SquidNet user interface settings are stored. Each submitting node should point to the same configuration path to avoid having to setup the same settings more than once. Example: Set the configuration for the master node. sqn --configPathSet \\RAID-SERVER00\volume_1\Squidnet\SquidNetConfig -node master Example: Set the configuration for all client nodes. sqn --configPathSet \\RAID-SERVER00\volume_1\Squidnet\SquidNetConfig -node clients
connectToMasterAs
Connect to master node as a "client" or "slave". Use this command to connect a local render node to the master. Example: Connect the local node to the master as a client. sqn --connectToMasterAs client
csv
Display output in a comma separated values (CSV) Example: Display job queue contents in csv format. sqn --jobQueueShow --csv Example: Display render farm nodes in csv format. sqn --renderFarmShow --csv
curCPUSpeed
Returns the current CPU speed for the specified render nodes. If the "--node" option is not specified, the default is to return the CPU speed for the local node. Example: Get current CPU speed for all slave nodes. sqn --curCPUSpeed --node slaves --delay 1
debug
Used for internal testing. Example: Display available options. sqn --debug help
137
SquidNet Network Render Manager
Example: Queue up 1000 verification jobs for testing. sqn --debug simulate --startFrame 1 --endFrame 1 --max 1000 --delay 1ms Example: Queue up 1000 verification jobs for testing but don't show any output. sqn --debug simulate --startFrame 1 --endFrame 1 --max 1000 --delay 1ms -quiet
delay
Used to periodically repeat the execution of a command. Append “ms” for millisecond resolution. Example: Display the output of the job queue every 5 seconds sqn --jobQueueShow --delay 5 Example: Display render farm status every 8 seconds. sqn --renderFarmShow --delay 8 Example: Execute a job and jobslice performance test every 100ms sqn --debug jobPerfTest --delay 100ms
enableSSA
Enable alternate SSA account for render jobs (Windows ONLY) Example: Submit a verification job but use specified SSA account credentials. sqn --netVerTest --enableSSA --usernameSSA "SquidNet Admin" --passwordSSA ********
emailQueueClear
Clear master server's email send queue. Example: sqn --emailQueueClear
emailQueueShow
Display content of master's email send queue. Example: sqn --emailQueueShow
emailSenderIdGet
emailSenderIdSet
emailBCCGet
Get the name of the currently configured email noticification sender id Example: Get sender id for email notifications. sqn -- emailSenderIdGet Set the name of the currently configured email noticification sender id Example: Set sender id for email notifications. sqn -- emailSenderIdSet "ABC Render Service" Show the list of currently set email BCC recipients. Example: Get current list of BCC recipients used on email notifications. sqn -- emailBCCGet
emailBCCSet
Set the list of email BCC recipients. Example: Set BCC recipients email notifications. sqn -- emailBCCSet "
[email protected],
[email protected],
[email protected]" To clear BCC list use “clear” keyword: sqn --emailBCCSet clear
138
SquidNet Network Render Manager
emailLogoImageGet
Displays the current email image logo URL settings Example: Show the current email image logo URL settings. sqn -- emailLogoImageGet
emailLogoImageSet
Also, requires image width and height fields.
Sets the image logo used in email notifications. Example: Set email notification image logo. sqn --emailLogoImageSet http://www.render4you.eu/images/banners/render4yousqn.jpg --imageHeight 150 --imageWidth 339 To reset logo to default settings, use “clear” keyword: sqn --emailLogoImageSet clear --imageHeight 0 --imageWidth 0
emailFooterGet
Display the current email footer message Example: Show the current email footer message. sqn -- emailFooterGet
emailFooterSet
Set the message that appears at the bottom of email notifications. Example: Set the message that appears at the bottom of email notifications. sqn --emailFooterSet "Render For You. Copyright Render For You, Inc. All rights reserved." To reset footer to default message, use “clear” keyword: sqn -- emailFooterSet clear
exceptionsClear
Clear all render farm exceptions Example: Clear all exceptions sqn --exceptionsClear
exceptionsShow
Displays all render farm exceptions Example: Displays all exceptions. sqn --exceptionsShow
execCmd
Execute script on specified nodes. Example: Run the specified script on all Windows nodes and show script output. sqn --execCmd S:\logs\setup-node.bat --node windows --execCmdShow Example: : Run the specified script on all Linux nodes. sqn --execCmd S:\logs\setup-node.sh --node linux --execCmdShow Example: : Run the specified script on all Mac OS nodes. sqn --execCmd S:\logs\setup-node.sh --node osx --execCmdShow Example: In a mixed render farm, execute a BASH script on Mac OS and Linux render nodes. sqn --execCmd $XPATH(SCRIPTS)\test.sh --node linux sqn --execCmd $XPATH(SCRIPTS)\test.sh --node osx Note: The"SCRIPTS" translation path points to a shared network folder.
139
SquidNet Network Render Manager
Example: : Run the specified script on nodes RENDERNODE12 and RENDERNODE13 sqn --execCmd S:\logs\setup-node.bat --node RENDERNODE12, RENDERNODE13 execCmdShow
Show the output from executed script. Example: Run the specified script on all Windows nodes and show script output. sqn --execCmd S:\logs\setup-node.bat --node windows --execCmdShow
help
Display sqn options. Example: sqn --help
jobHistoryClear
Clears job history on master node. Example: sqn -- jobHistoryClear
jobHistoryShow
Show job history. The job history is a listing off jobs that have been processed. Example: sqn -- jobHistoryShow
jobInputQueueShow
Shows the contents of the masters job input queue. Example: sqn --jobInputQueueShow
jobQueueClear
Removes (cancels) all entries in job queue. Warning: Use this command with caution. Job queue entries are not recoverable. Example: Clear entire job queue. sqn --jobQueueClear
jobQueueShow
Shows job queue listing. Example: Display contents of job queue. sqn -- jobQueueShow Example: Display contents of job queue but don't show column header. sqn --jobQueueShow --noHeader Example: Show all available columns. sqn --jobQueueShow --columnShow Example: Output job queue contents on CSV format sqn --jobQueueShow --csv Example: Display contents of job queue every 5 seconds.
140
SquidNet Network Render Manager
sqn --jobQueueShow --delay 5 Example: Display specific columns for job queue listing. sqn --jobQueueShow --column "Job Id",Frames,Slices,Owner --noHeader jobRemoveCompleted
Remove any completed jobs from job queue. Example: sqn --jobRemoveCompleted
jobRequeue
Restart completed job. When restarted, all job slices will b reprocessed. Example: Restart job id "#1542-A" and reprocess all job slices. sqn --jobRequeue #1542-A
jobRestartError
Restart a job that's in an error state. Only applies to jobs in an error state. Example: Restart job id #1542-E sqn -- jobRestartError #1542-E
jobSliceCacheShow
Displays contents of job slice cache. The job slice cache a s list of all recently accessed job slices. The list is maintained in memory on the master node and is periodically flushed to disk to conserve memory resources. Example: Display contents of job slice cache. sqn --jobSliceCacheShow
jobSliceEnvLog
Used with the "--submit" command to dump render node system environment variables to application's log file. Example: sqn --submit --template 3dsmax --appPath $APP(3DSMAX-64BIT) --jobId maxJob#1 --sceneName Shot_001 --sceneFile "$XPATH(SQUIDDIR)\\3DSMaxProjects\Scenes\AdvancedLighting\radiosity.max" --createOutputDir --outputDir $XPATH(SQUIDDIR)\Images\radiosity1 -imagePrefix test_ --imageFormat "PNG Image File" --imageSize "VGA (640 x 480)" --priority 1 --startFrame 1 --endFrame 20 --framePerSlice 1 --useAllCores -pools A --poolsAlternate B --notifyOptions all --include --processingNodes rendernode12 --onCondType requeue --onCondString "Error opening scene file:" --jobSliceEnvLog
jobSliceLogShow
Display application log for selected job slices. Example: Show applications logs for slices 1,2 and 3. sqn --jobId #1542-A --jobSliceLogShow 1,2,3
jobSliceRequeue
Requeue job's selected job slices. Example: Requeue job slices 1,2 and 3 for job id #1542-A sqn --jobId #1542-A --jobSliceRequeue 1,2,3
141
SquidNet Network Render Manager
jobSliceShow
Displays job slice list for selected job id. Example: Display job slices for job id #1542-A sqn --jobId #1542-A --jobSliceShow
killServer
Stop SquidNet server on specfied node(s) after a short delay Example: Kill server on node HELENA after a 20 second delay. sqn --killServer 20 --node HELENA
licenseInUse
Displays all used licenses a the nodes that are using them. Example: Displays allocated licenses. sqn --licenseInUse
licenseRegister
Register license from a text file. Example: Register license file. sqn --licenseRegister license.txt Example: When setting maser node, also register license. sqn --masterSet thor --licenseRegister license.txt
licenseShow
Displays all registered licenses. Example: Display all registered licenses. sqn --licenseShow
localConfig
Displays local configuration settings. Example: Display local configuration settings. sqn --localConfig
lock
Lock specified nodes to prevent them from processing jobs. If nodes are in the process of rendering jobs, locking is done after the job completes. Example: Lock nodes RENDERNODE10 and RENDERNODE11 sqn --lock --node rendernode10,rendernode11 Example: Lock all client nodes sqn --lock --node clients Example: Lock slave nodes sqn --lock --node slaves Example: Lock all Windows nodes sqn --lock --node windows Example: : Lock all nodes sqn --lock --node all Example: : Lock all Linux nodes sqn --lock --node linux
lockNow
Locks specified nodes and cancel any currently processing jobs. Example: Immediately lock all nodes.
142
SquidNet Network Render Manager
sqn -lockNow--node all loginShow
Displays logged in users on specified nodes. Example: Displayed all logged in users on Windows nodes. sqn --loginShow --node windows
mappedPathShow
Displays mapped drives on local node. Example: Display mapped drives on local node. sqn --mappedPathShow
masterProcess
Determine if master can render jobs. Valid Boolean values include: true, false, yes,no,1,0,enable,disable. Example: sqn --masterProcess true
masterSet
Sets the specified node as the master and resets all other render farm nodes to SLAVES. Before using this command, make sure to clear job queue first by using "-jobQueueClear" command. Example: Set THOR as the master node and register license at the same time. squidnet --masterSet THOR --licenseRegister license.txt
max
Used to by some command to specify a maximum value. Example: sqn .... --max 12 ....
min
Used to by some command to specify a minimum value. Example: sqn .... --min 32 ....
mkDir
Create a directory tree Example: Create a directory tree under using" SQUIDDIR" translation variable. sqn --mkDir $XPATH(SQUIDDIR)\dir1\dir2\dir3
netQueueShow
Displays a list of nodes that are actively processing jobs. Example: Display a list of nodes that are working on jobs. sqn --netQueueShow
netVerTest
Submits a SquidNet Verification test job. A SquidNet Verification job is just a job that tests out basic render farm functionality including: Verifying master slice distribution. Application launch and execution. Client and slave slice completion and response. Example: Submits a SquidNet Verification test job. sqn --netVerTest
node
Used to pass node names to CLI commands.
143
SquidNet Network Render Manager
Here are the possible options: Option all master clients slaves windows osx linux
Description Apply command to all render farm nodes. Apply command to master node only. Apply command to all client nodes. Apply command to all slave nodes. Apply command to all Windows nodes. Apply command to all Mac OS nodes. Apply command to all Linux nodes. Apply command to all nodes that belong to specified pools. Apply command to specified node names.
Example: Display information about all slave nodes sqn --nodeInfo --node slaves nodeInfo
Displays node information. If "--node" is not specified, defaults to local server node. Example: Display information about local server. sqn --nodeInfo Example: Display information about all slave nodes sqn --nodeInfo --node slaves Example: Display information about nodes THOR and HELENA sqn --nodeInfo --node thor,helena
nodeTypeSet
node-type “--node” option must also be specified.
noHeader
Set the type of SquidNet node. Options can be one of “master”, “client”, “slave” or “peer”. Example: Set node RENDERNODE 12 to be a slave. sqn –nodeTypeSet slave –node RENDERNODE12 Prevents the display of column headers. Example: Show contents of job queue but don't display column header. sqn --jobQueueShow --noHeader
noWait
Don't wait for command to complete. Instead, return control back to console as soon as command is queued. Example: Create MPEG4 videos with specified frame-rate, resolution and aspectratio. sqn --video S:\Images\flames --videoOutputDir S:\Archives\Videos -videoFrameRate 29 --videoResolution 800x600 --aspectRatio 4:3 --codec mpeg4 --noWait
optionsFile
Specify options in text file. This is an alternate method of specifying command line options. Example: Specify request options in text file. sqn --submit --template 3dsmax --optionsFile 3dsmax.txt Contents of optionsLightWave.txt options file: --appPath $APP(3DSMAX-64BIT) --jobId maxJob#1
144
SquidNet Network Render Manager
--sceneName Shot_001 --sceneFile "$XPATH(SQUIDDIR)\\3DSMaxProjects\Scenes\AdvancedLighting\radiosity.max" --createOutputDir --outputDir $XPATH(SQUIDDIR)\Images\radiosity1 --imagePrefix test_ --imageFormat "PNG Image File" --imageSize "VGA (640 x 480)" --priority 1 --startFrame 1 --endFrame 20 --framePerSlice 1 --useAllCores --pools A --poolsAlternate B --notifyOptions all --include --processingNodes rendernode12 --onCondType requeue --onCondString "Error opening scene file:"
Here is the same command with all options specified on command line: sqn --submit --template 3dsmax --appPath $APP(3DSMAX-64BIT) --jobId maxJob#1X --sceneName Shot_001 --sceneFile "$XPATH(SQUIDDIR)\\3DSMaxProjects\Scenes\AdvancedLighting\radiosity.max" --createOutputDir --outputDir $XPATH(SQUIDDIR)\Images\radiosity1 --imagePrefix test_ -imageFormat "PNG Image File" --imageSize "VGA (640 x 480)" --priority 1 --startFrame 1 --endFrame 20 --framePerSlice 1 --useAllCores --pools A --poolsAlternate B --notifyOptions all --include -processingNodes rendernode12 --onCondType requeue --onCondString "Error opening scene file:"
Example: Specify request options in text file. sqn --submit --template LightWave --optionsFile optionsLightWave.txt Contents of optionsLightWave.txt options file: --appPath $APP(LW11) --jobId Lightwave Scene 001 --startFrame 1 --endFrame 5 --framePerSlice 1 --priority 19 --configFolder \\raid-server00\Volume_1\SquidNet\LightWaveProjects\LW10\Config --contentFolder \\raid-server00\Volume_1\SquidNet\LightWaveProjects\Except_GiChapel --sceneFile \\raid-server00\Volume_1\SquidNet\LightWaveProjects\Except_GiChapel\Scenes\Chapel_v002.lws --username MyFirstName --comment Test scene: Lightwave Scene 001 --notes Some notes for this job --errsBeforeStop 20 --outputDir \\RAID-SERVER00\Volume_1\SquidNet\LightWaveProjects\Except_GiChapel\Images --imagePrefix chapel_ --imageSize SVGA (800 x 600) --imageFormat LW_PICT32(.pct) --processingNodes RENDERNODE11,RENDERNODE12 --compAction remove
Here is the same command with all options specified on command line: sqn --submit --template LightWave --appPath $APP(LW11) --jobId Lightwave Scene 001 --startFrame 1 -endFrame 5 --framePerSlice 1 --priority 19 --configFolder \\raidserver00\Volume_1\SquidNet\LightWaveProjects\LW10\FromRalph--contentFolder \\raidserver00\Volume_1\SquidNet\LightWaveProjects\Except_GiChapel --sceneFile \\raidserver00\Volume_1\SquidNet\LightWaveProjects\Except_GiChapel\Scenes\Chapel_v002.lws --username MyFirstName --comment Test scene: Lightwave Scene 001 --notes Some notes for this job --errsBeforeStop 20 --outputDir \\RAID-SERVER00\Volume_1\SquidNet\LightWaveProjects\Except_GiChapel\Images -imagePrefix chapel_ --imageSize SVGA (800 x 600) --imageFormat LW_PICT32(.pct) --processingNodes RENDERNODE11,RENDERNODE12 --compAction remove
pack
and other options.
Archive the input folder into the specified output file. Use the "--unpack" option to extract output files.
"--outputFile" option is mandatory
Other options include: Pack options Description --compress Compress output file to create a *.tar.gz file --password Encrypt output file with specified password. Output file will be *.tar.gz.enc --noWait Queue request and return to shell --node Send request to specified nodename Example: Archive S:\Images\0030-XSI-Fire-Ring folder to create E:\temp\0030XSI-Fire-Ring.tar archive file.
145
SquidNet Network Render Manager
sqn --pack S:\Images\0030-XSI-Fire-Ring --outputFile E:\temp\0030-XSI-FireRing.tar Example: : Archive and compress S:\Images\0030-XSI-Fire-Ring folder to create E:\temp\0030-XSI-Fire-Ring.tar.gz archive file. sqn --pack S:\Images\0030-XSI-Fire-Ring --outputFile E:\temp\0030-XSI-FireRing.tar --compress Example: Archive, compress and encrypt S:\Images\0030-XSI-Fire-Ring folder to create E:\temp\0030-XSI-Fire-Ring.tar.gz.enc archive file. sqn --pack S:\Images\0030-XSI-Fire-Ring --outputFile E:\temp\0030-XSI-FireRing.tar --compress --password my-password Example: Archive, compress and encrypt S:\Images\0030-XSI-Fire-Ring folder to create E:\temp\0030-XSI-Fire-Ring.tar.gz.enc archive file but don't wait for request to complete and process on node "thor" sqn --pack S:\Images\0030-XSI-Fire-Ring --outputFile E:\temp\0030-XSI-FireRing.tar --compress --password my-password --noWait --node thor password
Used with "--pack" option to encrypt output file. Example: Archive, compress and encrypt S:\Images\0030-XSI-Fire-Ring folder to create E:\temp\0030-XSI-Fire-Ring.tar.gz.enc archive file. sqn --pack S:\Images\0030-XSI-Fire-Ring --outputFile E:\temp\0030-XSI-FireRing.tar --compress --password my-password
passwordSSA
Password for alternate SSA account. (Windows ONLY) Example: Submit a verification job but use specified SSA account credentials. sqn --netVerTest --enableSSA --usernameSSA "SquidNet Admin" --passwordSSA ********
pidChildShow
Display all children of specified process id (PID) Example: sqn --pidChildShow 5768
pingTest
Connects to specified node and remove connection time. If "--node" is not specified, defaults to local server node. Example: Do ping test on local server. sqn --pingTest Example: Do ping test on all slave nodes sqn --pingTest --node slaves Example: Do ping test on nodes THOR and HELENA sqn --pingTest --node thor,helena
poolShow
render_file(maya)-->dev_dev01_wip_v11.mb --jobId GOGO-dev_dev01_render.1-24.55139 --sceneName dev_dev01 --submit --frameCheck --checkOutputContent --compAction after --compActionHrs 72 --createOutputDir --workDir "/theark/proj/local/generic/vfx/_assets/dev/dev01/squidnet" --startFrame 1 --endFrame 24 --stepFrame 1 --notes "/theark/proj/local/generic/vfx/_assets/dev/dev01/maya/scenes/wip/dev_dev01_wip_v11.mb" --outputDir "/theark/proj/local/generic/vfx/_assets/dev/dev01/rndr" --additionalFlags "--render --logdir "/theark/proj/local/generic/vfx/_assets/dev/dev01/tmp/logs/GOGO-55139.#.log/GOGO-dev_dev01_render.1-24.55139.log" /theark/proj/local/generic/vfx/_assets/dev/dev01/tmp/55139-dev_dev01-gogo_v01.py $(S)-$(E)"
Submitting Nuke composition job from Mac OS sqn --template generic --projectName generic --appPath /theark/bin/GOGO --jobDesc dev_dev01-gogo_v01.py-->comp_nuke->dev_dev01-comp_v11.nk --jobId GOGO-dev_dev01_comp.1-24.55139 --sceneName dev_dev01 --submit --frameCheck -checkOutputContent --compAction after --compActionHrs 72 --createOutputDir --workDir "/theark/proj/local/generic/vfx/_assets/dev/dev01/squidnet" --dependentJobList "GOGO-dev_dev01_render.1-24.55139" -startFrame 1 --endFrame 24 --stepFrame 1 --notes "/theark/proj/local/generic/vfx/_assets/dev/dev01/nk/dev_dev01comp_v11.nk" --outputDir "/theark/proj/local/generic/vfx/_assets/dev/dev01/comp/dev_dev01-comp_v11" --additionalFlags "--comp --logdir "/theark/proj/local/generic/vfx/_assets/dev/dev01/tmp/logs/GOGO-55139.#.log/GOGO-dev_dev01_comp.124.55139.log" /theark/proj/local/generic/vfx/_assets/dev/dev01/tmp/55139-dev_dev01-gogo_v01.py $(S)-$(E)"
Submitting mkmovie() job from Mac OS sqn --template generic --projectName generic --appPath /theark/bin/GOGO --jobDesc dev_dev01-gogo_v01.py-->mkmovie->dev_dev01-gogo_v01.py --jobId GOGO-dev_dev01_mkmovie.1.55139 --sceneName dev_dev01 --submit --frameCheck -checkOutputContent --compAction after --compActionHrs 72 --createOutputDir --workDir "/theark/proj/local/generic/vfx/_assets/dev/dev01/squidnet" --dependentJobList "GOGO-dev_dev01_render.1-24.55139 GOGO-dev_dev01_comp.1-24.55139" --startFrame 1 --endFrame 1 --stepFrame 1 --additionalFlags '--mkmovie --logdir "/theark/proj/local/generic/vfx/_assets/dev/dev01/tmp/logs/GOGO-dev_dev01_mkmovie.1.55139.log" /theark/proj/local/generic/vfx/_assets/dev/dev01/tmp/55139-dev_dev01-gogo_v01.py
15.4 Transactions 15.4.1.1 GUI 15.4.2 Command Line CLI Command
191
SquidNet Network Render Manager
Examples
192
SquidNet Network Render Manager
Chapter 16 FAQS
193
SquidNet Network Render Manager
15. FAQS 16.1 Configuration Questions 16.1.1 How do I relocate the master node? 16.2 Licensing 16.2.1 What types of licenses does SquidNet require? SquidNet provides the following types of licenses:
SquidNet Deluxe license (free): o Most advanced GUI features are disabled. o CLI features are disabled. o Includes 8-core render license. SquidNet Professional license: o One license is required per render farm (per master) o Unlocks advanced GUI features. o Unlocks CLI features. SquidNet CORE licenses: o Allows master to distribute jobs to more render nodes. o An 8-core license is provided with SquidNet Deluxe version.
16.2.2 Do SquidNet license ever expire? It depends on how they were purchased:
Permanent licenses never expire and are good for all future upgrades. Renewal (yearly, monthly, etc…) licenses are valid until expiration date.
16.2.3 Are SquidNet license good for all future updates? Yes, licenses are valid for all major and minor versions of SquidNet.
16.3 Installation 16.4 Configuration 16.4.1 Why is a render node’s state set to "Multiple IP Detection" ? A render node’s state is changed to "Multiple IP Detection" whenever the local server detects multiple network interface addresses. Whenever multiple network interfaces are detected, SquidNet cannot determine which one to use for network communications. To resolve this, on the master node GUI, select the node or nodes in question and open the “Network Address Select” then select the correct IP address to use.
194
SquidNet Network Render Manager
16.5 Job Submission 16.6 Job Management 16.6.1 Is the 'invalid character string detected'…'string too long'… message in the job log a problem? "invalid character string detected" means that a non-acscii character was detecter in the workflow. Make sure that no internationalized characters are present in any of the job request fields. SquidNet only supports standard US characters (A-Z,az, etc...)
16.6.2 Why do job slices get requeued when a low memory condition is detected? By default, memory checking is enabled in the preferences dialog. When enabled, a job slice is automatically requeued whenever memory usage on a specific node exceeds 97% (the default) of available memory. When detected, only nodes that have more than the max memory detected will be able to process that job slice.
16.7 Render Farm Management 16.7.1 What is the easiest way to remove a local node from rendering part of a job? In the render farm view, highlight the nodes in question and select one of the following options: 1. 2.
Lock node: Wait until the current job slice completes then lock the node. Lock Immediatily: immediatily requeue any active job slices and lock the node.
“Locking” a node prevents it from receiving any more job slices from the master. Use the “Unlock” option to add the node back into the rendering workflow.
195
SquidNet Network Render Manager
Chapter 17 G
TROUBLESHOOTIN
196
SquidNet Network Render Manager FAQs
16. TROUBLESHOOTING 17.1 Installation 17.2 Configuration 17.3 Job Submission 17.4 Job Management 17.5 Farm Management 17.6 Networking 17.7
197
SquidNet Network Render Manager
Chapter 18 APPENDICES
198
SquidNet Network Render Manager
Appendices
17. APPENDICES
199
