Muster Console Reference
Interface components
The following section will give a brief over the interface components provided by Muster Console, the complete real time graphical front end to the Muster Dispatcher Service.
The Muster Console window has a menu , a toolbar and five view types:
- The instances/host views
- The job queue views
- The log views
- The submission views
The Muster menus offer the same functionalities of the icons in the toolbar but divided in logical groups:
The file menu has the following options:
- Connect/Disconnect – Shows the connection dialog or terminates an active connection
- Preferences – Opens the Console preferences window
- Import - Imports a queue snapshot into the Dispatcher queue
- Export – Exports a snapshot of the current queue
- Quit – Exits from the Console application
The management menu has the following options:
- Configure Dispatcher – Opens the Dispatcher configuration dialog that lets you change the Dispatcher server options
- Configure Users – Opens the Users configuration dialog to manage the Dispatcher server internal users database
- Configure Pools - Opens the Pools configuration dialog to manage the Pools database.
- Configure Paths – Opens the Substitution paths configuration dialog that lets you change the paths used to apply the substitutions between different hosts and platforms.
- Configure Mails – Opens the mail configuration dialog that lets you define distribution lists to be used with the Dispatcher mailing notification system
- Edit templates – Opens the templates editing dialog that lets you change and propagate the templates on the fly
- Change Engine status – Starts and stops the Dispatcher internal engine
- Soft-restart Dispatcher – Perform a soft restart of the Dispatcher service forcing a reload of the configuration and a reconnection with the slaves
The action menu has the following options:
- Change password – Shows a dialog that lets you change the password for the currently logged user
- Register host(s) – Shows a dialog that lets you add a new host to the hosts/instances views
- Scan for hosts – Tells the Dispatcher to start a scan and find unregistered hosts in the current hosts/instances views
The view menu has the following options:
- Resync view – Reloads the selected views
- Create new view – Creates a new view and associate it to the current workspace
- Dispatcher remote logs – Opens a dialog that let you browse the logs on the Dispatcher
- Statistics view – Opens the statistics view dialog that lets you browse the Dispatcher statistics in real time and eventually export them
- History view – Opens the history view dialog that lets you browse the history of the Dispatcher activity and eventually, export and import the data.
The help menu has the following options:
- Help – Launches Acrobat Reader and opens the user’s manual
- Licensing – Opens a dialog that lets you change the current license
- Buy online – Connects to the Virtual Vertex store
- Request a license – Connects to the Virtual Vertex license requesting page
- Virtual Vertex site – Connects to the Virtual Vertex home page
- Check for a newer release – Check if a newer release is available online
- About - Shows the about screen
The instances/hosts view
The next figure shows a typical hosts/instances view in instances view mode. The view shows the connected and unconnected slaves instances, and their relative status will be immediately visible during their activity.
Each column of this view shows a specific property of the connected instance. The columns headers can also be used to sort the information. Just click on one of them and the view will be resorted. The default sorting is priority based.
The meaning of each column follows:
- Instance name: This is the name assigned to the machine. It can be the Netbios name on Windows or the name returned by gethostname() on Unix machines. When multiple instances are started on the client, an instance number is appended to the standard name. More information on instances are contained in the render client reference section. In addition to the host name, the icon on the left shows the current status of the machine. It can assume the following states in instance view mode:
The instance is idle and waiting for jobs
The instance is paused
The instance is busy processing a job
The instance is disconnected
The instance is connected but unavailable by configuration rules
The instance is connected but reserved by a logged user
or the following states in hosts view mode on the host nodes:
Console is actually directly connected to the host
Console is not connected to the host
Console is attempting a connection to the host
- IP address: This is the IP address of each machine. Multiple instances of the same client will share the column value
- Platform: Operative system running on the client
- Memory usage: This is the actual usage of RAM and swap file on the host
- Cpu usage: This is the actual usage of processors on the host
- Procs: This is the number of virtual processing units on the host. Hyper threading machines report multiple virtual units
- S: This is used in host view mode only and reports the status of the Render client service on the host. It can assume the following states:
Service is up and running
The Service status has not been queried yet
Service is stopped
The service is starting
You do not have the rights to query the remote service
Service is either uninstalled or an unknown error happened querying the service
- T and J: If you see a red cross in this column, it means that one or more templates or jobs have been add to the client exclusion list:
The exclusion list prevents further assignment of the job to the client to avoid infinite loops on faulty jobs - L: Shows the status of the real time log. When it’s enabled, you’ll see a log icon in this column
- ID: This is the internal identification number assigned by the Dispatcher to each host.
- Process priority: This is the scheduling priority for each process launched on the host.
- Priority: This is a priority number assigned to each instance. When idle, instances with the higher priority will be selected first in the job assignment logic.
- Status: Shows a textual status of the instance
- Current job: If busy, this field shows which job is currently assigned to the instance
- Current chunk: If busy with a multi frame job, this column shows the current job’s chunk
- Notes: Shows notes assigned to the instance
- Mac Address: Shows the MAC address of the NIC of the node. If you have multiple NICs installed, you’ll see multiple addresses separated by a pipe (|)
- Command line: If busy, shows the command line sent to start the current process
If you want to filter the view contents, you can use the options on the filtering bar:
- Network flow: Enable or disable data flow to the view. If you disable the network flow, each host view will be disabled and you’ll reduce the amount of network traffic between the Dispatcher and the Console. You should always disable a certain view network flow if you don’t need the windows to be constantly updated
- View mode: Changes between hosts and instances view mode
- Platform: Filters the contents by looking the instance’s platform
- Status: Filters the contents with the status value
- Pool: Filters the contents showing only instances belonging to a certain pool
Settings of the filtering bar will be stored persistently if you’re working on a custom workspace. If you’re working with the default, the settings will be reset on the next session. Check the workspaces section to learn more
Remember to disable the network flow on views you’re not interested in. This will reduce the amount of traffic between Console and the Dispatcher!
Managing the hosts/instances
By right-clicking on a host or instance in the instances view, you get a popup menu that allows you to perform different actions. The same applies to groups if you select multiple items:
This is a brief explanation of each option:
- Refresh: Forces a refresh of the instance’s status
- Pause: Pauses the instance
- Resume: Resumes a paused instance
- Kill and go on: Aborts a working instance, put the chunk back in the queue and move to the next or the same chunk, depending on clients availability and priority position
- Kill and redo: Aborts a working instance and restarts the chunk from the beginning
- Kill and pause: Aborts a working instance and pauses it
- Process priority menu: Sets the system process scheduling priority for the running process
- Set notes: Sets custom notes on the instance
- Real time log streaming: Enables or disables real time log streaming from the instance.
- Workstation logs: Accesses the workstation logs inspector. Through the inspector you can access logs produced by the instance, read and remove them
- Alarms: If you have an alarm running, you can reset the status from this menu
- Purge templates exclusion list: If you have some templates into the instance’s exclusion list, you can see and remove them from the associated menu
- Purge jobs exclusion list: If you have some jobs into the instance’s exclusion list, you can see and remove them from the associated menu
- Select job in exclusion list: Let you directly select a job fro the exclusion list
- Supported templates: Opens a dialog that shows the instance’s supported templates (engines)
- Soft restart: Performs a soft restart reloading the configuration, disconnecting and reconnecting it again. Unlike Reinit, this is done on any instance sharing the same node
- Configuration menu: Access the configuration option for the node
- Service: Accesses services query options
- System events: Sends system events like shutdown or restart to the node
- Remote control: Attempts to take control of the node using Remote Desktop, ssh or a valid tool. This may be customized in the Console preferences dialog
- Details: Give a full resume of the host details like the OS, the system capabilities (RAM and CPU) and the free space on the hard drives
- Remove: Removes the instance from the list. Works only on disconnected instances
Before attempting to configure automatic wakeup for the hosts, try to shutdown and wakeup an host manually using the options in the System Events menu. Also check the documentation of your motherboard and its Bios to understand if your system actually supports wake up on lan (Magic packet)
If you select the Configure menu, you’ll get the following additional popup:
The menu let you add or remove an host on the fly from a specific pool as well as inspecting configurations, copy configurations and create configuration presets. You can create several presets for each platform, and manage them using the configuration presets dialog:
Configuring the hosts
Accessing the configuration menu clicking on Configuration → Configure, the Console attempts a direct connection to the host to configure its behaviours. Once a successfully connection is made, you’ll be prompted with the configuration dialog as follows:
- Number of instances: Number of instances to spawn on service boot
- Default instances priority: The priority of the instances by default. Nodes with an higher priority will be allocated first. You can override the priority on an instance basis using panel below
- Overrides window: You can create several overrides on an instance basis as well as setting a specific process affinity mask for each instance
- Default child process priority: This defines the system default scheduling policy for each process spawned by the client. Values map to OS specific values
- Process start timeout: This defines a timeout value while Muster tries to catch the rendering process PID. If the timeout expires, the process is terminated and the chunk requeued and/or reported as failed
- Start instances in paused status: This starts the client in a paused status. Keep in mind that you should keep this feature disabled if you’re going for a fault tolerance, automated render farm. Having this option enabled, will prevent client activity after a forced reboot until an administrator resumes it
- Terminate processes tree: When the client tries to terminate a running process, the kill command is sent to the entire process tree (if available)
- Change priority on processes tree: When the client tries to change the process priority, the command is sent to the entire process tree (if available)
- Abort tasks on connection drop: Tells Muster to abort any running process if the socket TCP/IP connection between the client and the Dispatcher drops
- Suppress logs storage and parsing: If you want to skip the error processing and the log parsing, enable this function. This should be used for debug purposes only
The logs section let you configure the storage path for the activity and processes logs. You can also configure the following parameters:
- Severity level: This applies to the activity logs only and set the logs severity filter
- Destination folder: Sets the local destination folder to be monitored by the logs engine
- Clear logs older than: Specifies if you want to automatically delete logs files older than a certain number of days
- Clear older logs when the size exceeds: Specifies if you want to automatically delete logs files when their size exceeds a certain amount of Megabytes
- Rescan logs file system path every: If you store the logs on a common location and you remove or change the files outside of Muster, you can tell each client to rescan the folder to have them in sync with the Workstation log’s view
- Dispatcher address: The IP used to connect to the Dispatcher service
- Dispatcher listening port: The TCP/IP port where the Dispatcher is listening for incoming Render client connections
- Connection resolution: Attempts to reconnect after a disconnection when the amount of seconds is passed
- Client management port: The TCP/IP port where the Render client are listening for management connections incoming from Consoles
- Heartbeat in seconds: Send a pulse to the Dispatcher to let it know the client is alive. Depending on this setting, and the one in the Dispatcher global preferences, a client may be disconnected and flagged as offline if there’s no activity for a certain amount of time
- Enable management password: If you want to block incoming management connection on a client to prevent unauthorized changes, just put a password in this field and future connections will ask it to the users
- Broadcast presence: If you want the client to be auto discoverable by the Dispatcher on the network, flag this option. After you successfully configure your render farm, you should disable this option to avoid packet storming on your network
You can also specify a different username and password pair to be used when launching external processes. Unless you’ve a particular reason , you should leave those fields blank and relay on the configuration of the Services on Windows and the startup scripts on Unix.
The Engines section of the client configuration dialog let you configure variables required by each batch render template. The values are template specific but there’s always a variable pointing to the batch render executable. You can change this value to use different versions of your software and you can check/uncheck the checkbox near the template name to enable or disable the support of that particular engine.
This Window let you specify one or more substitution paths to be used when dealing with the client. If you’re configuring a Windows client, you can setup static drive mappings too. Having a drive mapped in your interface does not propagate the setting to Muster. As specified in the beginning, Muster lives in its own user address space. That means you’ve to tell it the drives to map.
By changing the way the client maps the drives let you keep under control the amount of connections to your file server limiting the amount of client licenses required. You can also activate the automatic drive mapping for Windows that embeds and automap a network share within each job.
Drive mappings information is embedded in the jobs only when submitting them from a Windows workstation and picking up the file from the drive map itself. There’s no way to embed a drive mapping information when submitting the job from Linux or MAC OS X. In a mixed OS environment, you should relay on static drive mappings configuration and disable this function.
The availability rules define when a particular client is available for rendering. You can choose its default availability by checking or unchecking the Available by default, tell the client if it has to abort the running process or wait its termination when its availability change by checking or unchecking the Wait tasks termination on status change and configure specific time lapses by clicking the Add rule button.
You can also tell the client to be available or not available depending on the presence or the absence of a particular process. This is very useful to let Muster co exists within other software that requires full control of the host.
Keep in mind that rules are always evaluated with other configurations like being available while a user is logged or during the screen saver activity. To make a client eligible for being available, the entire set of rules must be satisfied.
The shutdown section let you configure rules to automatically shutdown an host. Using the wake up feature, you can setup your render farm to shutdown on idle timing and wake up on demand when the full power is required. This is a great feature to reduce costs.
- Shutdown action: You can choose if the shutdown action is effectively a full host shutdown, a sleep action, or a restart. This is particular useful on Mac OS X, where there’s no way to wake up a powered off Mac. You can put it into sleep instead. During this phase, the wakeup function through the LAN will work
- Shutdown disabled with logged users: This tells Muster to avoid a shutdown process if someone is logged on the workstation
- Track remote accesses as local loggings: If someone is logged though a remote connection, Muster considers them as local loggings and prevents a shutdown if enabled in the previous option
- Shutdown the host when idle for: Tells Muster to shutdown the host after a certain amount of minutes of idle status
- Shutdown the host when paused for: Tells Muster to shutdown the host after a certain amount of minutes of paused status
- Scheduled shutdown: Defines custom rules to shutdown the host
- Matches shutdown rules with availability rules: Apply the action only if the availability rules are also valid
The wake up dialog let you configure specific rules to wake up an host through the Magic Packet (formerly Wake up on Lan). Apart from specific timings you can check the following options:
- When the farm is at full load: Wakeup the client when there are no idle hosts on the farm
- When the client is required by a job pool: Wakeup the client when it is required by the job’s pool
- Matches wakeup rules with availability rules: Apply the action only if the availability rules are also valid
The queue view
The next figure shows the jobs queue view. As you can see, jobs can be arranged in a hierarchical way. By dragging and dropping jobs, you can parent them to a new folder. To move them upward and downward in the queue you must change their priority.
Each column of this view shows a specific property of the job. The columns headers can also be used to sort the information. Just click on one of them and the view will be resorted. The default sorting is priority based.
As you can see each folder can contain multiple jobs. For viewing purposes they can be collapsed or expanded by clicking on the arrow on the left of the job status icon.
Jobs are arranged on a priority basis. This means that the jobs with the higher priority will be the first to be sent to the available instances.
When a job is a child of a folder it inherits some properties from its parent. In the specific, if the parent folder has a certain priority, even if the job has an higher one, it will be started only when the parent priority matches the current queue status. The priority of the job is valid when compared to the jobs belonging to the same parent. The same concept applies to the job destination pool. If it’s set to target the entire farm but its parent specifies a specific pool, the job will override its setting and will be sent to the pool specified by the parent folder. The Console will show this with a pool name called “Derived from parent”. If you select a different pool for the job, the derivation will be overridden.
An explanation of each column follows:
- Job name: This specifies the job name as well as its current status. In a similar fashion to the clients queue, the icon on the left of the job name can assume 10 different states:
The folder is currently paused. The setting propagates to the childs.
The folder is active
The job is on the queue and ready to start
The job is paused. Until you resume it, it will never start
The job is in progress
The job is in progress but reported some kind of warnings
The job has been completed but reported some kind of warnings
The job is in progress but reported some kind of errors
The job has been completed but reported some kind of errors
The job has been completed successfully
The job is locked. No operations are allowed on it until it’s unlocked
The job is archived. It won’t be visible until you change the view’s filters
The job is running a pre/post job action or the image assembler - L: Shows a lock when the job is locked
- T: The job has a timed action (like pause or resume at a certain time)
- ID: The ID is an internal progressive number assigned to each job by the dispatcher. You should take care about this number when setting job’s dependencies
- Project: Specify a custom string related to the job project
- Department: Specify a custom string related to the job department
- Priority: This is the priority for each job. The priority decides where a job is located inside the queue
- Destination Pools: This field shows the destination pool for the job.
- Excluded Pools: This field shows the excluded pools for the job.
- Re queued: At the end of each job, Muster can optionally check for missing frames. In case of a failure, one of more fractions of the job may be re queued. The re queued jobs will increase this counter. You can limit how many times a job or its fractions are re queued to avoid infinite loops for a faulty job.
- Engine: This field shows the targeted engine for the relative job
- Started on: For a started job, this field shows the exact time when the job has started
- Progress: For an in progress job, the field shows in percentage, the overall progress of the job. The progress does not include the in-rendering packets but it’s computed only on completed packets. Depending on your settings, the progress bar may show the status for each chunk in a visual way
- Completed on: For a completed job, this field shows the exact time when the job has been completed. In case of an in progress job, the field shows the estimated ending time
- Working nodes: The field shows how many instances actually are used to process the job. An instance can include one or more processors belonging to a particular render client. This depends on render client configuration
- Submission time: The field shows the time when the job was submitted
- Belonging to: The field shows the name of the user that submitted the job
- Starting frame: The field shows the starting frame of the job where applicable
- Ending frame: The field shows the ending frame of the job where applicable
- Total frames: Shows the total number of jobs frames or the number of the slices (where applicable)
- Filename: The field shows the primary filename of the job (where applicable)
- Notes: Show additional notes for the job
If you want to filter the view contents, you can use the options on the filtering bar:
- Network flow: Enable or disable the flow of data to the view. If you disable the network flow, each host view will be disabled and you’ll reduce the amount of network traffic between the Dispatcher and the Console. You should always disable a certain view network flow if you don’t need the data shown to be updated
- Engine: Filters the contents by showing only jobs for certain templates
- Status: Filter the contents by showing only jobs with a certain status
- User: Filter the contents by showing jobs belonging to a certain user
At the end of the filter bar, there are two buttons. The first one lets you create a new folder, while the second one lets you filter the contents of the view by searching a particular string in the jobs names.
Settings of the filtering bar will be stored persistently if you’re working on a custom workspace. If you’re working on the default, the settings will be reset on the next session. Check the workspaces section to learn more
Remember to disable the network flow on views you’re not interested it. This will reduce the amount of traffic between Console and the Dispatcher!
Managing the jobs
By right-clicking on a job in the queue view (or making a multiple selection), you get a popup menu that lets you take control of one or multiple jobs:
This is a brief explanation of each option:
- Expand all: Expands the entire queue tree
- Collapse all: Collapses the entire queue tree
- Expand branch: Expands the entire tree selected branch
- Collapse branch: Collapses the entire tree selected branch
- Move item(s) to root: Moves the selected item(s) to the root level
- Move item(s): Moves the selected item(s) inside the selected folder
- Pause: Pauses a job. A paused job won’t be sent to clients for processing.
- Resume: Resumes a paused job
- Time actions: Pause or resume a job at a certain time
- Reinit: Reset the job to its initial state. If you have instances working on the job, the processes will be terminated
- Nodes working on: Perform actions on the instances currently working on the job
- Open output folder: If the job specifies an output folder, the command opens the folder to view the files
- Chunks details: Opens the chunks detail window
- Manual frame check: Opens the manual frame check dialog
- Rename: Renames the job
- Set project: Sets the unique job project string. This may be used later for filtering and arranging the jobs
- Set department: Sets the unique job’s department string. This may be used later for filtering and arranging the jobs
- Set priority: Sets the job priority. Jobs with an higher priority will be processed first. Jobs that share the same priority are sorted by the ID number. Smaller IDs will be sent first
- Set maximum instances: Set the maximum number of instances allowed to work simultaneous on the job
- Set dependencies IDs: Set a list of jobs dependancies. If a job depends from another job, it will wait until it completes before being eligible for processing
- Change depend mode: Change the result mode required by the dependancies
- Set pool(s): Change the destination pools of the job
- Set excluded pool(s): Change the excluded pools of the job
- Set owner: Change the job owner
- Set notes: Set the custom notes on the job
- Overrides: Let you change the jobs behaviours overrides
- Set template: Change the job current template
- Inspect drive mappings: Let you check, if available, the drive mappings embedded in the job
- Lock: Locks the job. Any operation is forbidden until the job is unlocked
- Unlock: Unlocks a locked job
- Delete: Removes the job from the queue
- Delete requeued jobs: Removes the jobs requeued from the selected one
- Send job to: Send job properties to one of the available submission views of your workspace
The log view
The log pane is the most important Muster window. Messages reported by render clients or status/error messages reported by the Dispatcher are displayed here. You should pay attention to its output to be able to track error and/or render failures. The log shows the time the event has occurred, the text of the event, the user that thrown that event (Service refers to Dispatcher Service), the machine originating the event and its relative IP. The lines have different colours according to the kind of the event:
- The log entry is an error. This may come from an instance activity or a dispatcher notice.
- The log entry is a successfully event. This may come from an instance activity or a dispatcher notice.
- The event comes from an user action
- The event comes from a system action (Dispatcher)
If you want to filter the view contents, you can use the options on the filtering bar:
- Network flow: Enable or disable the data flow to the view. If you disable the network flow, each host view will be disabled and you’ll reduce the amount of network traffic between the Dispatcher and the Console. You should always disable a certain view network flow if you don’t need windows to be constantly updated
- Source: Filters the contents showing logs entries originating from a certain entity
You can clear the current log content by clicking the Clear log button. If you clear the log, it will be done on the Dispatcher side meaning that each Console will get its log cleared.
The submission view
The Submission view allow you to check and edit existing jobs properties and submit new jobs. The job properties list is dynamic, and changes according to the selected job and its properties template. At the top of the view, a combo box lets you choose which view automatically fills the list with the selection. The buttons at bottom of the view allows to submit a new job, edit an existing one, save and load presets stored in XML based files, or access the presets manager that stores easily recallable settings from a persistent list.
The properties on the submission view depends on the selected engine but they share some common ones explained below:
General section
- Job name: Specifies the name for your new job or the ones you’re editing
- Engine: This drop-down combo box shows the available render engines. By selecting one of them the submission view will update its fields reflecting the engine template properties
- Job project: Specifies the project of the job (custom string)
- Job department: Specifies the department of the job (custom string)
- Job status: Overrides the default status in the queue
- Job priority: Specifies the priority level for the job. Jobs with higher values will be processed first
- Include Pool(s)/Hosts: Specify one or more render pool to use for the render, you can also specify single hosts
- Exclude Pool(s)/Hosts: Specify one or more render pool to exclude from the render, you can also specify single hosts
- Depend from job ID(s): If you want to prevent the job processing until one or more jobs are completed, write their IDs there (comma separated)
- Dependence mode: Specify what kind of results Muster should expect from dependent jobs to allow the job processing
- Notes: You can put any kind of notes inside this field
- Stars on: Specify a starting time for the job
- Minimum logical units: Specify a minimum amount of virtual processors an host must have to be eligible to process the job
- Minimum cores: Specify a minimum amount of physical cores an host must have to be eligible to process the job
- Minimum physical cpus: Specify a minimum amount of physical processors an host must have to be eligible to process the job
- Minimum cores speed: Specify a minimum speed in GHz an host must have to be eligible to process the job
- Minimum ram amount: Specify a minimum amount of Ram, in Megabytes, an host must have to be eligible to process the job
- Minimum disk space: Specify a minimum amount of free space on the physical disks, in Megabytes, an host must have to be eligible to process the job
Multiframe options section
This section is available only for multi frame jobs. It contains the following fields:
- Packet size: This field specifies the size in frames for each chunk sent to a client. You should carefully balance this value carefully depending on your scene frame range and the processing power of your clients.
- Maximum instances: This field specifies the maximum number of render instances used by the job. A value of 0 instructs the job to use all the available instances.
- Check for missing frames: This field activates the automatic missing frames feature available on the Dispatcher. At the end of the job processing, the Dispatcher will scan for missing frames and requeue them automatically. Additional information can be found in the Missing frames section
- Image name prefix: If you use the missing frames features, you need to specify here the expected filename prefix as well as the frames path
Image slicing options section
This section is available only for single frame jobs. It contains the following fields:
- Frame number: Put here the frame to render as a single sliced image
- Number of slices: This value specifies the number of separate slices to create (basically it means how many machines will render the frame). At the end of the rendering, the slices will be reassembled by the Dispatcher
- Image width: Specify the width of the final image
- Image height: Specify the height of the final image
- Aspect ratio: Specify the aspect ratio of the final image. This value applies only to certain engines
- Anti alias overlap: The anti aliasing filters may work incorrectly on the image borders. To provide slices that may produce a valid result, you may need to increase the amount of pixels rendered on the border of the slices. Increasing this value will generate images that are a bit larger but correctly reassembled. Additional pixels are rendered on the slices but the final image will reflect the expected size.
- Maximum instances: This field specifies the maximum number of render instances used by the job. A value of 0 instructs the job to use any available instances
Broadcast options section
This section is available only for broadcast jobs. It contains the following fields:
- Packet type: Single instance/Every instance, by choosing this field you can tell Muster to send the job to every running instance, or only one process for each IP address.
Single host options section
This section is available only for single host jobs. It contains the following fields:
- Delegated host: You can specify the instance name or IP address of the host to be used.
Overrides
- Process valid exit codes: Overrides the expected valid exit codes (comma separated)
- Process warning exit codes: Overrides the expected warning exit codes (comma separated)
- Process error exit codes: Overrides the expected error exit codes (comma separated)
- Valid process log’s texts: Overrides the expected string or regular expression to match valid log’s texts
- Warning process log’s texts: Overrides the expected string or regular expression to match warning log’s texts
- Error process log’s texts: Overrides the expected string or regular expression to match error log’s texts
- Chunks timeout: Overrides the job chunks timeout
- Chunks maximum requeue: Overrides the maximum number of times a chunk can be requeued
- Mail address: Overrides the destination email address to send notifications
- Override job notifications: By checking this field, you can override the default notification event related to jobs notifications
- Override chunks notifications: By checking this field, you can override the default notification event related to chunks notifications
Actions
- Action: Specifies the executable to launch as a pre or post job/chunk action
- Check return code: Tells Muster to check for the action return code and abort processing according
- Override timeout: Overrides the default timeout for the action
Managing submission presets
The submission panel lets you store and manage properties presets. Preset are retrievable through a pop-up menu and are stored on a template basis:
If you click the Add preset menu entry, the current values of the submission dialog are stored as a new preset and you’re prompted for the preset’s name. By clicking the Manage presets menu entry, you can Rename, Delete or Duplicate existing presets.
Presets are always stored on a template basis, there’s no way to duplicate a preset done for a different template than the one it belongs to
Managing workspaces and views
Muster Console supports customizable workspaces. A workspace contains a set of views and their settings, and stores them persistently across different sessions.
You can access the workspaces functions by clicking the icon next to the workspaces selector combo box in the right corner of the menu bar.
By using the menu entries, you can Duplicate, Rename and Delete the current workspace. If you want to create a new workspace, you must start duplicating current; this will inherit your current settings and will create the new workspace. Once you create a new workspace, it will be available in the workspaces list:
Muster Console supports three default workspaces. You cannot Delete or Rename them, but you can Duplicate them to originate new workspaces.
The settings of the views like their columns ordering and filtering are stored with the workspace. If you’re working on the default, you’ll lose your settings each time you exit from the Console. Always create your own workspace if you want to keep your settings!
From the view menu, you can create a new view selecting the Create new view submenu. Once you create a new view, you can manage the state of it using the buttons at the top of it:
The buttons let you rename, hide or destroy a view. Remember that once you hide a view using the central button, you can recall it using the views popup menu, right clicking on a view header or on the empty workspace area of Console (if no view is visible). This is a typical views popup menu you may get:
Customizing a view
If you want to customize the columns of a view, you can right click on its headers and invoke the popup menu:
If you click the Customize headers entry, you’ll get the following dialog:
The customize view headers window lets you choose exactly which column you want to show, its position in the list, its content alignment and the default column size.
Customizing the view’s columns and the view’s filters is a good way to have multiple views with different contents available in the workspace.
If you’re working on a custom workspace, the settings changed by this window will be stored persistently with the view
Muster Console preferences
The Muster Console can be configured selecting Preferences from the menu or clicking the relative button on the toolbar. The general section has the following options:
- Check for updates on startup: Tells the Console application to connect to the Virtual Vertex site checking for updates each time you start it
- Automatically check services status on startup: When active, the Console will attempt to query the service status of any host available in the hosts views. This may be network and resource consuming, so depending on the size of your farm, you may need to disable this and check the services statuses manually when required
- Management port: This configures the management port used to connect to the Dispatcher service
- Keep management connections always active: When you configure a client, a direct connection is established to get its properties and to send the configuration back. If you check this option, the connection is persistent
- Listen on broadcast from clients to grab Cpu and Ram usage: If status broadcast is enabled on a render client , console checks for this traffic and updates the usage counters according
- Show floating point numbers for frames: If you’re going to work with floating point frames numbers, you may need to activate this option for a correct feedback of the frames numbers
- Paint detailed job progress arrays: Paints a progress bar that reflects the status of each chunk. This may be resource intensive, so depending on the size of your queue, and the number of the jobs, you may need to deactivate this option
- Calculate progresses for folders: Calculates the folders progresses by averaging the progress of the childs
- Ask confirmation on unsafe operations: If activated, when you make an unsafe operation, like deleting a job in the queue, Muster Console will show a confirmation dialog
- Show notes on instances as popups: If active, when you hover with your mouse on an instance icon you’ll see a popup with the host’s notes
- Show notes on jobs as popups: If active, when you hover with your mouse on a job icon you’ll see a popup with the job’s notes
- Queue interactive search: When changed the search filter into the queue, Muster recalculates the result on each keystroke, this may be cpu intensive when dealing with big queues
- Hosts views double click action: Choose an action to perform when you double click on a hosts/instance
- Jobs view double click action: Choose an action to perform when you double click on a job
- Chunks views double click action: Choose an action to perform when you double click on a chunk
- Load frame range from Lightwave’s scenes: If you’re working with Newtek Lightwave, you can tell Console to load the frame range from the selected .LWS file when you pick one from the submission view.
- Textual contents: Defines an external viewer used to open textual contents
- Visual contents: Defines an external viewer used to open image files
The jobs section lets you define several default values for the submission views:
The remote control section lets you define how to remote control an host when you select the Remote Control function from the popup menu:
The default values on Windows use Remote Desktop to access Windows hosts, and Putty to access Unix hosts through an ssh connection while Unix’s Console uses Vnc or Ssh. You should configure the command line according to your environment and your preferred software.
Particular attention should be paid to the Unix services control. While Windows is able to query remote services using its built-in API, we rely on a modified version of Plink (part of the Putty suite) to access Unix hosts through an SSH connection and check the Services remote status. You should not change the command line preconfigured unless you’ve particular needs. Just be sure to specify a root username and password in the preferences and open the SSH port on the remote hosts.
While you can use the Windows API and Plink from a Windows hosts running Console to query any platform service status, there’s no way to query a Windows service status from a Linux or Mac OS X platform!
The alarms section let you define local alarms thresholds on the hosts
The graphs colours section lets you define the maximum amount of statistics hold in memory by the Console and the colours used by the various graphs:
The skin section let you modify or define new skins to be used by Console. The syntax of the skinning language is based on the QT CSS cascading style sheets. For further information on how to modify the skins, please reefer to the QT4 CSS style documentation available on the QT website at qt.nokia.com
If you need to reset the Console preferences or store them for future reference, they can be found into the registry for Windows (HKEY_CURRENT_USER\Software\Virtual Vertex\Muster\8\Console) and inside your home folder (.Muster) on Linux and Mac OS X
The Chunks detail
The Chunks detail is a very useful summary window that lets you manage packets (called chunks) that build a job. On multi frame and image slicing jobs, each chunk is a logical representation of a processing phase. On broadcast jobs, each chunk is dedicated to an instance or an host. Single host jobs do not have chunks.
This is a typical view of the chunks detail window:
Each packet is numbered with its internal ID and reports the following information:
- Absolute Frames: Those are the effective frames of your sequence chunk. They may not reflect the number appended on the frame name.
- Relative Frames: Those are the frames numbers that will be generated. They may not reflect the absolute frames but will match with the final files. The first packet reflects the value of the starting frame value of a job.
- Status: This field reports the current status of the chunk. It can assume 4 different state and reflects the colour of the icon on the left of the chunk’s ID. Possible states are: On hold, In progress ,completed or completed with warnings.
- Requeued: When a chunk is re-queued using the automatic missing frames feature, this field will be creased each time the process occurs.
- Starting time: This field shows the starting time when a chunk has been submitted to a client. It remains blank for a “On hold” packet.
- Ending time: This field shows the ending time for a completed chunk.
- Time span: This field shows the effective amount of time taken by a completed chunk for its processing.
- Rendered by: This field shows the name of the instance that has rendered the chunk. If chunks are flagged as completed by an user, the field shows “Forced by console”.
- Results: If a packet completed with some kind of error, it will be reported inside this field otherwise the field will stay blank.
To manage chunks , right click on them and you’ll get the following pop-up menu:
Operations permitted by this menu are simply the requeuing of a completed chunk or the setting of the chunk in a completed status. When forcing a chunk to a different status, render clients that are currently rendering the chunk may receive an abort message.
If you select the Open chunk’s log item, the logs of the hosts that processed the chunk is opened, and the related file selected automatically.
A special type of chunk is the Image assembler one. This is a chunk that’s always found at the end of a single frame job. It’s included inside the chunk view only for consistency but actually rendered by the Dispatcher service when the previous packets have been completed. Its purpose is to assemble slices created by the instances.
The chunks view reports just the chunks of the current job status, if you want to compare the entire lifecycle of the job, you can click the “Gather chunks history” button and have a tree based view with the results of each chunks in the history. This let you also check why a particular chunks has failed and also recover the logs of the failed or requeued chunks.
The frames check window
The missing frames window will pop up when you right click on a job and select “Manual frame check”:
The dialog lists the job frames grouped in a logical way. Each missing or good sequence of consecutive frames is reported with a summary of the average size in bytes. The frame parser can be configured to check only the existence of the files or to check the file size.
To start the frame check, just press the “Start frames check” button. Remember that, if you have not specified a frames path and prefix during job submission, you may need to specify them in the frames check dialog. The prefix of the file must be include the path and any character that precedes the number that identifies the actual frame number.
For example , if your final renders are saved inside the folder \\Myserver\myrenders and have a name similar to this:
Scene1_0001.tga Scene1_0002.tga Scene1_0003.tga
the prefix you’ve to supply is \\Myserver\myrenders\Scene1_ After running the frames check, you can automatically queue the wrong packets pressing the “Queue wrong packets” button or just queue the list’s selection using the “Queue selection” button. If you want to configure the parser, select Configure behaviours:
The parser can also check the files sizes between a minimum and a maximum value. This is particular useful if you are rendering frames in a compressed file format where file sizes can change between each frame.
If you disable the size parser, Muster will check only the frame existence.
Browsing the network statistics
Each time you open the Console, it receives some statistics from the Dispatcher on a regular basis. You can access such statistics through the Statistics view menu entry in the View menu:
You can use the button menu in the upper right corner of the window to export the current data. You can then reload the data using the History view explained later.
Managing the history
The history view can be used to retrieve the processing and the usage history of the Dispatcher itself, store queries and reload saved archives.
You can access the history browser by selecting History view from the View menu:
If you query the jobs history you’ll get a resume of the selected period, you can then export the data in a proprietary format, or to well known formats like PDF.