Template engine (Python)

Since version 8, Muster implements a brand new template engine. The template engine takes care of instruct Muster how to interact with external render engines or batch jobs. The new engine is entirely based on Python 3.3. Through the template engine you can:

  • Configure external applications
  • Build command lines
  • Create submission dialogs
  • Declare special actions
  • Create complex presets dialogs
  • Define the behaviours of clients
  • Create custom execution environments

The template engine expect a .py file for each template that should create a basic class derived from the standard MTemplateAPI.MTemplate class configured with custom capabilities.

A minimal template requires at least the following elements:

  • A class that inherit the base class MTemplateAPI.MTemplate
  • Configuration of the class default values like the template ID, the template name, the template logic (multiframe, image slicing, broadcast or target host)
  • Configuration of the template logics default values
  • Configuration of the template fields that will be mapped directly to a job attributes and will appear in the submission dialog
  • Configuration of the supported platforms (Win / Mac / Linux) specific values
  • Redefinition of the requires virtual functions that spawn the processes, build the command lines and check for the results

After creating the template class, it must be installed into the global template manager singleton using :

MTemplateAPI.MManager.getUniqueClass().installPyTemplate(myTemplateClass)

Each section may contain special values, stepping inside Muster default templates could be a good starting point in learning the template Python classes.

We’ll step across building a template for Maya rendering to learn more:
The first check you’ve to do when building a new template, is taking a look at the command line required by the engine, it our case, the basic syntax maya expect is:

Render –r render_type –proj project_directory –sf start_frame –ef end_frame –bf by_frame path_to_scene

Keeping that in mind, let’s build the template using the internal software rendering:

''' Import Muster base modules '''
import MClientAPI
import MTemplateAPI
 
''' Get the template manager singleton '''
manager = MTemplateAPI.MManager.getUniqueClass()
 
''' Define a new class derived from MTemplateAPI.MTemplate base class , override required methods '''
class MayaTemplate(MTemplateAPI.MTemplate):
    def __init__(self):
        MTemplateAPI.MTemplate.__init__(self)
	self.setID(1)
	self.setName("Maya Sw")
	self.setDescription("Maya software render")
	self.setTemplateLogic(MTemplateAPI.kTemplateMultiframe)
	self.setDefaultPriority(1)
	''' Multiframe template specific settings '''
	self.multiframe.setEnableFrameCheck(1)

As you can see, the first section declares the basic template parameters, giving a template name, description and unique identifier. The logic is set to Multiframe that requires a

self.multiframe.setEnableFrameCheck(1)

if you want to enable the frame check feature. This syntax shows how to access specific logics features.

The next section configures the submission dialog either for Console and the web server:

mayaFile = MTemplateAPI.MTemplateItemFile("MAYAFILENAME","Maya scene file name","Specifies the maya scene file to render","",0,1,1,"*.ma;*.mb")
mayaProject = MTemplateAPI.MTemplateItemFolder("MAYAPROJECTPATH","Project directory","Specifies the maya project path","",0,1,1)
mayaRenderPath = MTemplateAPI.MTemplateItemFolder("MAYADESTINATION","Frames destination","Specifies the frames destination","",1,1,1)
mayaFrames = MTemplateAPI.MTemplateItemFramesRange("MAYAFRAMES","Frames","Specifies the frames ranges for the rendering job in the 1-10x1s1n1,... form","1-10",0,1,1)
mayaDigits = MTemplateAPI.MTemplateItemInteger("MAYADIGITS","Digits","Specifies the number of digits for the frame numbering",1,0,0,1)
mayaDigits.setMinMaxValues(1,65535)
mayaAbortRender = MTemplateAPI.MTemplateItemYesNo("ABORTRENDER","Abort on log","Immediately abort the rendering if an error string is found during log output",0,0,0,1)
self.addSubmissionItem(mayaFile)
self.addSubmissionItem(mayaProject)
self.addSubmissionItem(mayaRenderPath)
self.addSubmissionItem(mayaFrames)
self.addSubmissionItem(mayaDigits)
self.addSubmissionItem(mayaAbortRender)
''' items mapping to Muster default tags '''
self.addMapping("MAYAFRAMES","frames_range")
self.addMapping("MAYAFILENAME","job_file")
self.addMapping("MAYAPROJECTPATH","job_project")
self.addMapping("MAYARENDERPATH","output_folder")

The previous code builds a set of fields object that are installed into the template using the addSubmissionItem() function call. Each object is a subclass of the MTemplateItem base class.

The addMapping() function let you specify which field maps to the required Muster internal fields like the start frame, end frame and so on..

''' Windows support '''
self.platformWindows.setPlatformEnabled(1)
self.platformWindows.setEnableErrorCheck(1)
self.platformWindows.setEnabledByDefault(1)
self.platformWindows.setFramesFloating(3)
self.platformWindows.setDetectionLogic(MTemplateAPI.kProcessChild)
self.platformWindows.setDetectionLogicProcessName("Mayabatch.exe")
applicationPath = MTemplateAPI.MTemplateItemFile("MAYAEXECUTABLE","Maya batch render executable","Path to Maya render.exe application batch render","c:\\program files\\Autodesk\\Maya 2018\\bin\\render.exe",0,0,1,"*.exe")
self.platformWindows.addClientConfigurationItem(applicationPath)
startingFolderPath = MTemplateAPI.MTemplateItemFolder("MAYASTARTINGFOLDER","Starting folder","Specify the starting folder for the render process","c:\\program files\\Autodesk\\Maya 2018\\bin",0,0,1)
self.platformWindows.addClientConfigurationItem(startingFolderPath)
processorUsage = MTemplateAPI.MTemplateItemCombo("PROCUSAGE","Processors usage","Specify the number of processors to use for each render instance","0",0,0,1)
processorUsage.addItem("All","0")
processorUsage.addItem("1","1")
processorUsage.addItem("2","2")
processorUsage.addItem("3","3")
processorUsage.addItem("4","4")
self.platformWindows.addClientConfigurationItem(processorUsage)

The previous code sample shows the engine declaration section for the Windows platform.
It not all sets platform specific attributes, but it also install a set of fields inside the platform itself. Those fields will be available into the client configuration dialog and can be accessed later to construct the command line as well as to provide additional behaviours.

def onBuildCommandLine(self, platform, job, chunk, clientTemplatePreferences,instanceNum):
	renderCmd = "-r sw"
	renderCmd += " -n " + clientTemplatePreferences['PROCUSAGE']
	renderCmd += " -proj \"" + job.attributeGetString(manager.resolveMappingToJob(self.getID(),"MAYAPROJECTPATH")) + "\""
	if job.attributeIsEnabled(manager.resolveMappingToJob(self.getID(),"MAYADESTINATION")):
		renderCmd += " -rd \"" + job.attributeGetString(manager.resolveMappingToJob(self.getID(),"MAYADESTINATION")) + "\""
	renderCmd += " -s " + '%.3f' % chunk.getStartFrame()
	renderCmd += " -e " + '%.3f' % chunk.getEndFrame()
	renderCmd += " -b " + '%.3f' % chunk.getByFrame()
	renderCmd += " -rfs " + str(chunk.getFrameStart())
	renderCmd += " -rfb " + str(chunk.getFrameStep())
	renderCmd += " -pad " + job.attributeGetString("MAYADIGITS")
	renderCmd += " " + job.attributeGetString("ADD_FLAGS")
	renderCmd += " \"" + job.attributeGetString(manager.resolveMappingToJob(self.getID(),"MAYAFILENAME")) + "\""
	return renderCmd

The previous code sample shows how to override a class virtual method to provide a specific function. In this case, we need to tell Muster how to build the final command line.

def onGetApplicationPath(self,job,clientTemplatePreferences,pathOut ):
	pathOut.setString(clientTemplatePreferences['MAYAEXECUTABLE'])
	return MTemplateAPI.MTemplateError()
 
def onGetApplicationStartingFolder(self,job,clientTemplatePreferences,pathOut):
	pathOut.setString(clientTemplatePreferences['MAYASTARTINGFOLDER'])
	return MTemplateAPI.MTemplateError()

In the previous code, we tell Muster how to find the command line and the starting folder for the batch job. In that case, we grab the values from the client configuration values using the clientTemplatePreferences supplied dictionary.

def onCheckLog(self,job,chunk,clientTemplatePreferences,log):
	platform = MClientAPI.GetPlatform()
	lines = log.split("\n")
	for idx, val in enumerate(lines):
		marker = manager.checkForLineParsingOverrides(job,chunk,val,idx)
		if marker.getType() != MClientAPI.MTextFileMarker.kFileMarkerNone:
			if marker.getType() == MClientAPI.MTextFileMarker.kFileMarkerError:
				errors.append(marker)
			elif marker.getType() == MClientAPI.MTextFileMarker.kFileMarkerWarning:
				warnings.append(marker)
			elif marker.getType() == MClientAPI.MTextFileMarker.kFileMarkerSilenceError:
				''' The builtin parser has a skip rule for this line, do not include it in the warnings/errors but in the silenced errors'''
				silencedErrors.append(marker)
			elif marker.getType() == MClientAPI.MTextFileMarker.kFileMarkerSilenceWarning:
				''' The builtin parser has a skip rule for this line, do not include it in the warnings/errors but in the silenced warnings'''
				silencedWarnings.append(marker)
			else:
				''' We have no fixed rule , parse the line according to the template logic '''
				mayaError = -1
				mayaWarning = -1
				length = -1
				mayaError = val.find("Error: ")
				mayaWarning = val.find("Warning: ")
 
				if mayaError == 0:
					length = len(val)-7;
					marker = MClientAPI.MTextFileMarker(MClientAPI.MTextFileMarker.kFileMarkerError,idx,7,length)
					errors.append(marker)
				if mayaWarning == 0:
					length = len(val)-9;
					marker = MClientAPI.MTextFileMarker(MClientAPI.MTextFileMarker.kFileMarkerWarning,idx,9,length)
					warnings.append(marker)
 
	if log.find("Total Time For Render") == -1:
		return MTemplateAPI.MTemplateError(3,"Missing \"Total Time For Render\" keyword. Batch render may have crashed",MTemplateAPI.MTemplateError.kTemplateErrorTypeError)
	if log.find("Total Elapsed Time For Maya") == -1:
		return MTemplateAPI.MTemplateError(3,"Missing \"Total Elapsed Time For Maya\" keyword. Batch render may have crashed",MTemplateAPI.MTemplateError.kTemplateErrorTypeError)
	if log.find("completed.") == -1:
		return MTemplateAPI.MTemplateError(3,"Missing \"completed.\" keyword. Batch render may have crashed",MTemplateAPI.MTemplateError.kTemplateErrorTypeError)
 
	if len(errors) > 0 and  len(warnings) > 0:
		return MTemplateAPI.MTemplateError(2,"Errors and warnings reported in the log. Please check the chunk details for a detailed list",MTemplateAPI.MTemplateError.kTemplateErrorTypeError)
	elif len(errors) > 0:
		return MTemplateAPI.MTemplateError(2,"Errors reported in the log. Please check the chunk details for a detailed list",MTemplateAPI.MTemplateError.kTemplateErrorTypeError)
	elif len(warnings) > 0:
		return MTemplateAPI.MTemplateError(1,"Warning reported in the log. Please check the chunk details for a detailed list",MTemplateAPI.MTemplateError.kTemplateErrorTypeWarning)
 
	return MTemplateAPI.MTemplateError()
 
def onCheckExitCode(self,job,chunk,clientTemplatePreferences,exitCode):
	if exitCode == 211:
		return MTemplateAPI.MTemplateError(exitCode,"Abnormal render termination",MTemplateAPI.MTemplateError.kTemplateErrorTypeError)
	if exitCode != 0:
		return MTemplateAPI.MTemplateError(exitCode, "%d Exit code different from expected 0 value" % exitCode,MTemplateAPI.MTemplateError.kTemplateErrorTypeWarning)
 
	return MTemplateAPI.MTemplateError()

The previous virtual functions tell Muster how to process the batch job output log as well as the process exit code.

''' Create an instance of the template class and install it into the template manager context '''
template = MayaTemplate()
manager.installPyTemplate(template)

The last code creates an instance of the class and installs it into the template manager.

Further information on the template syntax can be found looking into the API classes reference as well as into the existing templates.

Any template is able to override Muster default behaviours by redefining one or more virtual functions of the base class. This is a list of the available functions and their meaning as seen by the Python engine, also you should consider checking the C API class description for a complete description of the Template class.

Under the load column, you can find two values, normal and critical. When a function is flagged as critical, that means it's run in line with the module networking or cores functions. If you take too much time running your Python function, the module may became unresponsive. So pay particular attention.

Functions called on the engine template

Function name and parametersExpected return variableContextLoad
onCheckChanged(string fieldName,bool state, ref MTemplateItemsFields fieldsToChange)voidConsoleNormal
Invoked by Console when a template field activator changes, useful to automate fills of fields or change values in reaction to the click
onFieldChanged(string fieldName,string fieldValue, ref MTemplateItemsFields fieldsToChange)voidConsoleNormal
Invoked by Console when a template field value changes, useful to automate fills of different fields and/or invoke actions like scene parsing after the file selection
example
onBuildCommandLine(int platform, MJob job, MChunk chunk,ref stringMap clientTemplatePreferences, int instanceNum)stringRenderclientnormal
Invoked by the client to build the process command line
example
onGetFlagForRenderLayers(MJob job, layers)stringRenderclientnormal
Invoked by the client to get the required flag to render just a subset of render layers
example
onBuildAdditionalParameters(ref MPropertiesMap attributes)stringRenderclientnormal
Invoked by Console to build additional parameters for the command line
example
onGetApplicationPath(MJob job,MChunk chunk,ref stringMap clientTemplatePreferences, ref MStringRef pathOut)MTemplateError Renderclientnormal
Invoked by the client to get the process path
example
onGetApplicationStartingFolder(MJob job,MChunk chunk,ref stringMap clientTemplatePreferences, ref MStringRef pathOut) MTemplateErrorRenderclientnormal
Invoked by the client to get the process execution folder
example
onBuildEnvironment(MJob job, MChunk chunk,ref stringMap clientTemplatePreferences, ref MEnvironment existingEnvironment)MEnvironmentRenderclientnormal
Invoked by the client to build an environment block for the process
example
onGetWindowsBatFile(MJob job, MChunk chunk,ref stringMap clientTemplatePreferences, ref MEnvironment existingEnvironment)stringRenderclientnormal
Invoked by the client to parse a supplied bat file content to setup the environment
onCheckForSubframeAdvancingString(MJob job, MChunk chunk, string line)boolRenderclientnormal
Invoked by the client on each log line parsing to check for a frame completion string
example
onDetectRunningProcess(MPid mainProcess, vector of MProcessSnapshot hostProcesses, ref stringMap clientTemplatePreferences,ref MPid runningProcessPid)kTemplateFunctionBehaviourRenderclientnormal
Invoked by the client to detect the process effective running PID
onCheckForFramesMask(MJob job, MChunk chunk, string line, ref MStringRef maskOut)boolRenderclientnormal
Invoked by the client on each log line parsing to guess the frames path
example
onCheckForFramesMask(MJob job, MChunk chunk, string line, ref MStringRef maskOut,ref MStringRef layerOut)boolRenderclientnormal
Invoked by the client on each log line parsing to guess the frames path, used for multi layer render
example
onCheckForSubframeProgress(MJob job, MChunk chunk, string line, ref MStringRef progressOut)boolRenderclientnormal
Invoked by the client on each log line parsing to guess the percentage of the current frames
example
onCheckLogLine(MMJob job,MChunk chunk,ref stringMap clientTemplatePreferences, string line, int lineNum, list:MTextFileMarker warnings, list:MTextFileMarker errors, list:MTextFileMarker silencedWarnings, list:MTextFileMarker silencedErrors)MTemplateErrorRenderclientnormal
Invoked by the client for each log line, this is useful to abort the current render on certain log errors
example
onCheckLog(MJob job, MChunk chunk,ref stringMap clientTemplatePreferences, string log,list:MTextFileMarker warnings, list:MTextFileMarker errors, list:MTextFileMarker silencedWarnings, list:MTextFileMarker silencedErrors)MTemplateErrorRenderclientnormal
Invoked by the client to check the full log at the end of the render
example
onCheckExitCode(MJob job,MChunk chunk,ref stringMap clientTemplatePreferences, int exitCode)MTemplateErrorRenderclientnormal
Invoked by the client to check the process exit code
example
onApplicationFinder(ref MStringRef moduleRegExp, ref MStringRef& moduleTag)intRenderclientnormal
Invoked by the Services control panel applet to configure automatic application finding
example
onFindApplication(ref MStringRef basePath,ref stringMap clientTemplatePreferences)boolRenderclientnormal
Invoked by the Services control panel applet to find a specific application directly from the template
example
onModuleFound(ref MStringRef moduleExec,ref MStringRef modulePath,ref stringMap clientTemplatePreferences)boolRenderclientnormal
Invoked by the Services control panel applet when an application executable is found
example
onGetSlicesInputFilename(MJob job,MChunk chunk)stringRenderclientnormal
Invoked by image slicing jobs to know each chunk slice file path
example
onGetSlicesOutputFilename(MJob job, MChunk chunk)stringRenderclientnormal
Invoked by image slicing jobs to know the final assembled image name
example
onGetSliceBoundaries(MJob job,MChunk chunk,MTemplateSliceBoundaries boundaries)voidRenderclientnormal
Invoked by image slicing jobs to know how to expect slices and how to assemble them
example
onStartProcess(MJob job, MChunk chunk, int instanceNum, MPid& runningProcessPid))voidRenderclientnormal
Invoked by the client when a new process should be started
example
onTerminateProcess(MJob job, MChunk chunk, MPid process, int instanceNum))voidRenderclientnormal
Invoked by the client when a running process should be terminated
example
onCollectProcessOutput(MPid mainProcess, MPid runningProcess, int instanceNum, out MStringRef output))voidRenderclientnormal
Invoked by the client to collect the output of a spawned process
example
onChangeProcessPriority(MJob job, MChunk chunk, MPid process, int priority, int instanceNum))voidRenderclientnormal
Invoked by the client when the system priority of a process should be changed
example
onCheckProcessTermination(MJob job, MChunk chunk, MPid process, int instanceNum))voidRenderclientnormal
Invoked by the client to check if the process has been completed
example

Functions called on the engine template and the global template (0)

Function name and parametersExpected return variableContextLoad
onValidateJobSubmission(MJob job, ref MStringRef err)boolConsole/mrtoolcritical
Used to validate a job submission
example
onClientIdling()NoneRenderclientcritical
Invoked by the renderclient on idle phases
example
onCheckClientAvailability()kTemplateFunctionBehaviourRenderclientcritical
Invoked by the renderclient to pass back its availability status
example
onInstanceOverrideImpersonationUser(MJob job, MChunk chunk, clientTemplatePreferences, stringRef usernameOut stringRef passwordOut)kTemplateFunctionBehaviourRenderclientcritical
Invoked before a process is started to check if the template wants to override the impersonation user account
example
onDispatcherIdling()NoneDispatchercritical
Invoked by the Dispatcher on idle phases
example
onDispatcherOverrideImpersonationUser(MJob job, MChunk chunk, stringRef usernameOut stringRef passwordOut)kTemplateFunctionBehaviourDispatchercritical
Invoked before a process is started to check if the template wants to override the impersonation user account
example
onDispatcherOverrideFlagJobsAsArchived(MJob job)kTemplateFunctionBehaviourDispatchercritical
Overrides the flag jobs as archived function
example
onDispatcherOverrideDeleteArchivedJobs(MJob job)kTemplateFunctionBehaviourDispatchercritical
Overrides the automatic delete of archived jobs
example
onDispatcherOverrideBackupJob(MJob job)kTemplateFunctionBehaviourDispatchercritical
Overrides the automatic backup of jobs
example
onPrepareTasks(MJob job, list of MTemplateTask tasks)NoneDispatchercritical
Invoked after a multitask template job has been submitted
example
onClientConnected(MNode node)NoneDispatchercritical
Invoked when a a node/instance is connected
example
onClientDisconnected(MNode node)NoneDispatchercritical
Invoked when a a node/instance is disconnected
example
onClientDisconnectedByHeartbeath(MNode node)NoneDispatchercritical
Invoked when a a node/instance is disconnected by an heartbeat check failure
example
onClientStatusChanged(MNode node)NoneDispatchercritical
Invoked when a a instance status changes
example
onPauseJob(MJob job)NoneDispatchercritical
Invoked when a job is paused
onUserPauseJob(MJob job,MUser user)NoneDispatchercritical
Invoked when a job is paused by an user
example
onResumeJob(MJob job)NoneDispatchercritical
Invoked when a job is resumed
onUserResumeJob(MJob job,MUser user)NoneDispatchercritical
Invoked when a job is resumed by an user
example
onDeleteJob(MJob job)NoneDispatchercritical
Invoked when a job is deleted
onUserDeleteJob(MJob job,MUser user)NoneDispatchercritical
Invoked when a job is deleted by an user
example
onSubmitJob(MJob job)NoneDispatchercritical
Invoked when a job is submitted
onUserSubmitJob(MJob job, MUser user)NoneDispatchercritical
Invoked when a job is submitted by an user
example
onReinitJob(MJob job)NoneDispatchercritical
Invoked when a job is reinit
onUserReinitJob(MJob job, Muser user)NoneDispatchercritical
Invoked when a job is reinit
example
onEditJob(MJob job)NoneDispatchercritical
Invoked when a job is edited
onUserEditJob(MJob job, MUser user)NoneDispatchercritical
Invoked when a job is edited by an user
example
onJobPropertiesChanged(MJob job)NoneDispatchercritical
Invoked when a property of a job is changed
onJobPropertiesChanged(MJob job,MUser user)NoneDispatchercritical
Invoked when a property of a job is changed by an user
example
onJobStart(MJob job)MTemplateErrorDispatchercritical
Invoked when the job is started
example
onJobEnd(MJob job)MTemplateErrorDispatchercritical
Invoked when a job has completed its chunks
example
onJobCompleted(MJob job)MTemplateErrorDispatchercritical
Invoked when a job is effectively completed
example
onChunkStart(MJob job,MChunk chunk, ref stringMap clientTemplatePreferences,int instanceId)MTemplateErrorDispatchercritical
Invoked when a chunk is started
example
onChunkEnd(MJob job,MChunk chunk,ref stringMap clientTemplatePreferences, int instanceId)MTemplateErrorDispatchercritical
Invoked when a chunk is completed
example
onChunkSubmitted(MJob job,MChunk chunk, int instanceId)MTemplateErrorDispatchercritical
Invoked when a chunk is submitted. Fired from the Dispatcher
example
onChunkCompleted(MJob job,MChunk chunk, int instanceId)MTemplateErrorDispatchercritical
Invoked when a chunk is completed. Fired from the Dispatcher
example
onChunkRequeued(MJob job,MChunk chunk)MTemplateErrorDispatchercritical
Invoked when a chunk is requeued. Fired from the Dispatcher
example
onCustomNodeMessage(int instanceId, string message)NoneDispatchercritical
Invoked when an instance sends a custom message
example
onValidateJobDependancies(job)kTemplateFunctionBehaviourDispatchercritical
Invoked to validate a job selection
example
onValidateChunkDependancies(chunk)kTemplateFunctionBehaviourDispatchercritical
Invoked to validate a chunk selection
example
onMailJobCompleted(MJob job, framesHaveBeenRequeued, MStringRef subjectOut, MStringRef htmlBodyOut, MStringRef&plainBodyOut)boolDispatchercritical
onMailChunkCompleted(MJob job, MChunk chunk, MNode node, chunkHasBeenRequeued, chunkHasMissingFrames, MMStringRef subjectOut,MStringRef htmlBodyOut,MStringRef plainBodyOut)boolDispatchercritical
onMailSendLogs(list of MLogs, MStringRef subjectOut, MStringRef htmlBodyOut, MStringRef plainBodyOut)boolDispatchercritical
onMailConfigurationsChanged(message, MStringRef subjectOut, MStringRef htmlBodyOut, MStringRef plainBodyOut)boolDispatchercritical
onMailSoftRestarts(message, MStringRef subjectOut, MStringRef htmlBodyOut, MStringRef plainBodyOut)boolDispatchercritical
onMailEngineStatus(state, MStringRef subjectOut, MStringRef htmlBodyOut, MMStringRef plainBodyOut)boolDispatchercritical
onMailExclusionListEvent(message, MStringRef subjectOut, MStringRef htmlBodyOut, MStringRef plainBodyOut)boolDispatchercritical
onMailJobStart(MJob job, MStringRef subjectOut, MStringRef htmlBodyOut, MStringRef plainBodyOut)boolDispatchercritical
Invoked by the templates to build mailing notifications
example