Page 6 – Virtual Vertex

Also to complete the explanation, there’s another way to submit a job using the MClientAPI , but instead of importing the module inside the running Python environment, you would spawn MPython.exe (inside the Muster installation folder) that comes preconfigured, and pass your custom script that uses the MClientAPI functions as well as passing some parameters grabbed from Maya itself. If you need the power of Python to pre-flight your job, this may be another option you may consider.

Leonardo Bernardini

4th June 2019 at 6:24 pm #17694

This is a sample job JSON with all the attributes/overrides enabled, off-course you should use just the things you need:


{
    "jobName": "Untitled",
    "templateId": 1,
    "templateVersion": "",
    "parentId": -1,
    "dependIds": [
        "100"
    ],
    "dependMode": 0,
    "dependLinkMode": 0,
    "priority": 1,
    "project": "job-project",
    "department": "job-department",
    "camera": "job-camera",
    "shot": "job-shot",
    "sequence": "job-sequence",
    "includedPools": [],
    "excludedPools": [],
    "owner": "admin",
    "startOn": 12,
    "pauseOn": 0,
    "resumeOn": 0,
    "requeued": 0,
    "packetSize": 4,
    "chunksPriority": 0,
    "chunksInterleave": 2,
    "progress": 0,
    "emergencyQueue": false,
    "packetType": 1,
    "maximumInstances": 0,
    "assignedInstances": 0,
    "exitCodesErrorCheckType": 0,
    "logsErrorCheckType": 0,
    "overrideMailNotificationsAtJobLevel": true,
    "mailNotificationsAtJobLevelType": 1,
    "overrideMailNotificationsAtChunkLevel": true,
    "mailNotificationsAtChunkLevelType": 0,
    "overrideNotificatorNotificationsAtJobLevel": true,
    "notificatorNotificationsAtJobLevelType": 1,
    "overrideNotificatorNotificationsAtChunkLevel": true,
    "notificatorNotificationsAtChunkLevelType": 1,
    "overrideMobileNotificationsAtJobLevel": true,
    "mobileNotificationsAtJobLevelType": 1,
    "overrideMobileNotificationsAtChunkLevel": true,
    "mobileNotificationsAtChunkLevelType": 1,
    "overrideStartMailNotificationsAtJobLevel": true,
    "startMailNotificationsAtJobLevelType": 1,
    "overrideStartMailNotificationsAtChunkLevel": true,
    "startMailNotificationsAtChunkLevelType": 0,
    "overrideStartNotificatorNotificationsAtJobLevel": true,
    "startNotificatorNotificationsAtJobLevelType": 0,
    "overrideStartNotificatorNotificationsAtChunkLevel": true,
    "startNotificatorNotificationsAtChunkLevelType": 1,
    "overrideStartMobileNotificationsAtJobLevel": true,
    "startMobileNotificationsAtJobLevelType": 1,
    "overrideStartMobileNotificationsAtChunkLevel": true,
    "startMobileNotificationsAtChunkLevelType": 1,
    "overrideChunksTimeout": true,
    "chunksTimeoutValue": 320,
    "overrideValidExitCodes": true,
    "overrideWarningExitCodes": true,
    "overrideErrorExitCodes": true,
    "overrideValidExitCodesValue": "1",
    "overrideWarningExitCodesValue": "2",
    "overrideErrorExitCodesValue": "3",
    "overrideMinimumThreads": true,
    "overrideMinimumCores": true,
    "overrideMinimumPhysical": true,
    "overrideMinimumSpeed": true,
    "overrideMinimumRam": true,
    "overrideMinimumDiskSpace": true,
    "overrideMaximumChunksRequeue": true,
    "overrideMaximumChunksRequeueValue": 5,
    "overrideMinimumThreadsValue": 13,
    "overrideMinimumCoresValue": 14,
    "overrideMinimumPhysicalValue": 15,
    "overrideMinimumSpeedValue": 16,
    "overrideMinimumRamValue": 17,
    "overrideMinimumDiskSpaceValue": 18,
    "logsParsingRules": "",
    "updateTime": 1559665190,
    "attributes": {
        "ABORTRENDER": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "ADD_FLAGS": {
            "value": "-addflag",
            "state": true,
            "subst": true
        },
        "MAYADIGITS": {
            "value": "1",
            "state": true,
            "subst": false
        },
        "by_frame": {
            "value": "1",
            "state": true,
            "subst": false
        },
        "confirmed_mask": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "end_frame": {
            "value": "10",
            "state": true,
            "subst": false
        },
        "environmental_variables": {
            "value": "ENV=1",
            "state": true,
            "subst": false
        },
        "fc_check_file": {
            "value": "1",
            "state": true,
            "subst": false
        },
        "fc_check_image_dimensions": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "fc_file_high": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "fc_file_low": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "fc_image_height": {
            "value": "1080",
            "state": true,
            "subst": false
        },
        "fc_image_width": {
            "value": "1920",
            "state": true,
            "subst": false
        },
        "fc_open_image": {
            "value": "1",
            "state": true,
            "subst": false
        },
        "fc_recursion": {
            "value": "1",
            "state": true,
            "subst": false
        },
        "fc_render_layer_mode": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "frame_check": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "frames_mask": {
            "value": "QSBmcmFtZSBtYXNrCi9wYXRoL3RvL2ZyYW1lL21hc2sK",
            "state": true,
            "subst": true
        },
        "frames_range": {
            "value": "1-10",
            "state": true,
            "subst": false
        },
        "job_file": {
            "value": "/path/to/maya/scene.mb",
            "state": true,
            "subst": true
        },
        "job_project": {
            "value": "/path/to/maya",
            "state": true,
            "subst": true
        },
        "memo": {
            "value": "job-notes",
            "state": true,
            "subst": false
        },
        "movie_assembler": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "movie_assembler_framerate": {
            "value": "30",
            "state": true,
            "subst": false
        },
        "movie_assembler_input_flags": {
            "value": "",
            "state": false,
            "subst": false
        },
        "movie_assembler_output": {
            "value": "",
            "state": true,
            "subst": true
        },
        "movie_assembler_output_flags": {
            "value": "",
            "state": false,
            "subst": false
        },
        "movie_assembler_template": {
            "value": "48",
            "state": true,
            "subst": false
        },
        "output_folder": {
            "value": "/path/to/frames/destination",
            "state": true,
            "subst": true
        },
        "overridestatus": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "post_chunk_action": {
            "value": "/post/chunk/action.cmd",
            "state": true,
            "subst": true
        },
        "post_chunk_action_check_retcode": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "post_chunk_action_timeout": {
            "value": "20",
            "state": true,
            "subst": false
        },
        "post_job_action": {
            "value": "/post/job/action.cmd",
            "state": true,
            "subst": true
        },
        "post_job_action_check_retcode": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "post_job_action_timeout": {
            "value": "20",
            "state": true,
            "subst": false
        },
        "pre_chunk_action": {
            "value": "/pre/chunk/action.cmd",
            "state": true,
            "subst": true
        },
        "pre_chunk_action_check_retcode": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "pre_chunk_action_timeout": {
            "value": "20",
            "state": true,
            "subst": false
        },
        "pre_job_action": {
            "value": "/pre/job/action.cmd",
            "state": true,
            "subst": true
        },
        "pre_job_action_check_retcode": {
            "value": "0",
            "state": true,
            "subst": false
        },
        "pre_job_action_timeout": {
            "value": "20",
            "state": true,
            "subst": false
        },
        "start_frame": {
            "value": "1",
            "state": true,
            "subst": false
        }
    }
}

Leonardo Bernardini

4th June 2019 at 6:01 pm #17692

No, names are different, that’s why I suggested you to send a job through the Muster web interface with Google Developer Tools open in Chrome, and enable all the attributes. This will send a job with everything enabled, and you can have a base reference point for your JSON.
If you do not have confidence with Chrome and developer tools, I can provide you a base json with everything enabled.

Leonardo Bernardini

4th June 2019 at 4:27 pm #17688

Sure,

First of all the docs are at https://www.vvertex.com/wiki900/doku.php?id=muster:9.0:restful_http_api

Our Houdini-PDG integration script uses the REST APIs to submit a job through Muster, it is into the latest Muster distribution.
By the way, to have a fine control of what you’re sending to Muster, step 1 is to open the Muster web interface, open the developers tools in Chrome and inspect the POST call that’s sent from the web interface when you send a job, in that way you can debug the exact JSON required by your jobs/template submissions.

Then, as a general workflow:

1) In Python 2.x you can use urllib2 to make an http request, it’s a standard Python library and Maya carries it.
2) You also need to deal with JSON , so importing json is a good thing. Those are the bootstrap imports from our PDG submitter:

import os
import sys
import shutil
import logging
import ssl
import json
import re
import threading
import shlex
import platform

try:
    _create_unverified_https_context = ssl._create_unverified_context
except AttributeError:
    # Legacy Python that doesn't verify HTTPS certificates by default
    pass
else:
    # Handle target environment that doesn't support HTTPS verification
    ssl._create_default_https_context = _create_unverified_https_context
try:
    import urllib2 as urlreq # Python 2.x
except:
    import urllib.request as urlreq # Python 3.x

import urllib

3) Before each submission, you need to obtain an access token from the HTTP APIs, this can be done called /api/login/username=admin&password=password , the response is pure JSON and into the ResponseData , you need to grab the authToken. Little example:


url = self.getBaseDispatcherUrl() + "/api/login?" + urllib.urlencode({'username':dispatcher_login,'password':dispatcher_password})
req = urlreq.Request(url)
req.add_header('User-Agent', 'Houdini (PDG)')
contents = urlreq.urlopen(req,context=context).read()
res = json.loads(contents)
if 'ResponseStatus' in res:
    if 'Code' in res['ResponseStatus']:
        retCode = int(res["ResponseStatus"]["Code"])
        if retCode == 200:
            if 'ResponseData' in res:
                if 'authToken' in res["ResponseData"]:
                    self.authToken = res["ResponseData"]["authToken"]
                else:
                     logger.error("Authentication failed:{}:{}".format(retCode,res["ResponseStatus"]["Description"]))

Once you have an access token, you can submit . Also remember to logout to deallocate your session, or it will expire automatically after 20 minutes, but if you’re making massive submissions, logout is good

4) Submit the job, again this is a sample from PDG, you need to fine tune your job attributes as well as the parameters you’re sending:

 url = self.getBaseDispatcherUrl() + "/api/queue/actions?" + urllib.urlencode({'name':'submit','authToken':self.authToken})
            requestData = "{\n\"RequestData\":\n{\n\"platform\" : " + str(submission_platform) + ",\n\"job\":\n{"
            requestData += "\"jobName\":{},\n".format(json.dumps(job_name))
            requestData += "\"templateId\":{},\n".format(dispatcher_pdg_template_id)
            if len(dispatcher_pdg_template_version):
                requestData += "\"templateVersion\":{},\n".format(json.dumps(dispatcher_pdg_template_version))

            requestData += "\"priority\":{},\n".format(muster_job_priority)
            if muster_job_emergency == "on": requestData += "\"emergencyQueue\":1,\n"
            included_pools = filter(None, muster_job_pool.split(','))
            excluded_pools = filter(None, muster_job_ex_pool.split(','))
            encoded_included_pools = []
            encoded_excluded_pools = []
            for x in included_pools:
                encoded_included_pools.append(json.dumps(x))
            for x in excluded_pools:
                encoded_excluded_pools.append(json.dumps(x))

            if len(encoded_included_pools):
                    requestData += "\"includedPools\":[" + ','.join(encoded_included_pools) + "],\n"
            if len(encoded_excluded_pools):
                    requestData += "\"excludedPools\":[" + ','.join(encoded_excluded_pools) + "],\n"
          
            if self.parent_id > 0: requestData += "\"parentId\" : " + str(self.parent_id) + ",\n"
            requestData += "\"attributes\":{"
            requestData += "\"PDGEXECUTABLE\" : { \"value\" : " + json.dumps(executable) + " , \"state\" : true, \"subst\" : true }"
            requestData += ",\"PDGARGUMENTS\" : { \"value\" : " + json.dumps(arguments) + " , \"state\" : true, \"subst\" : true }"
            requestData += ",\"memo\" : { \"value\" : " + json.dumps(muster_job_notes) + " , \"state\" : true, \"subst\" : false }"
   
            requestData += "}\n"
            requestData += "}\n}\n}\n"
            req = urlreq.Request(url,requestData)
            req.add_header('Content-Type', 'application/json')
            req.add_header('User-Agent', 'Houdini (PDG)')
            contents = urlreq.urlopen(req,context=context).read()
            res = json.loads(contents)
            if 'ResponseStatus' in res:
                if 'Code' in res['ResponseStatus']:
                    retCode = int(res["ResponseStatus"]["Code"])
                    description = res["ResponseStatus"]["Description"]
                    if retCode == 200:
                        job_id = res["ResponseStatus"]["objectId"]
                        self.jobs_lock.acquire()
                        self.active_jobs[job_id] = { 'name' : item_name , 'index' : item_id }
                        self.jobs_lock.release()
                        logger.debug('Scheduling: {} {} successfully'.format(node_name, item_name))
                        work_item.data.setInt("muster_jobid", job_id, 0)
                        return scheduleResult.Succeeded
                    else:
                        logger.error("Scheduling failed: {}:{}".format(retCode,description))
                else:
                     logger.error("Scheduling failed: Invalid response from server")
            else:
                logger.error("Scheduling failed: Invalid response from server")

Hope this points you to the right direction. Cheers !

Leonardo Bernardini

4th June 2019 at 3:43 pm #17685

Hi Alex,

Mixing Python interpreters is always a bad idea. What happens here is that when you bring up inside the Maya Python 2.7.11 the MClientAPI module, it in turns bring ups its own 3.3 interpreter for its internal stuff that deals with the template engine. You should try to re-define also PYTHONHOME and PYTHONPATH pointing to the Muster installation directory. By the way, embedding the Muster API inside Maya/Nuke may be tedious also to maintain. Have you considered using the REST/APIs that’s the way we are going internally for new connectors, and basically let you do any thing you can do with the MClientAPI and in an easier way ? Or is there a particular function in the MClientAPI module you need, so I can suggest you a workaround ?

Leonardo Bernardini

23rd May 2019 at 7:55 pm #17504

Hi Alex,

You have a pletora of different ways to handle your scenario:

1) If you still want to submit a job using a command line tool (our mrtool cmd) and keep the current work flow as similar as the previous, you can directly pass a list of environmental variables using the mrtool flag (-attr) that let you pass a custom attribute to a job. Custom attributes are metadata that do not fit the standard jobs settings (like name, type, template etc etc). In that situation, the environment metadata is called “environmental_variables”. So the flag for mrtool may be something like -attr “environmental_variables” “value=1;value2=2” 0 . The last 0 means if you want to content of the attribute to be passed through the paths substitution engine, but I think that’s not your case so you can keep it to 0. Be aware also that inside environmental variables, you can use python inline evaluation (more on this on our wiki), so you can basically generate the environmental dynamically depending on the content of the MChunk or MJob object. If you want further info on this , ask in your reply.

2) If you want to really stay with your current workflow, I assume that when you submit a job, you create an xml, json or whatever else file side by side with the submission. You can still take this path, just be sure to first create the environmental file and then submit the job. In that case, I suggest you to create a custom template derived from the one you need, and as you already guess, create the environment inside the template onBuildEnvironment. That function pass you also the MJob object, so you can read the variables and/or attributes to get your original file location, calculate the location of your environmental variable file, parse it and change the environment according.

3) For a future proof solution by the way, I suggest you to submit the jobs using the REST APIs, that’s actually simpler than the tons of flags required by mrtool , or use a command line that triggers our Python interpreter (MPython inside muster installation folder) that in turns call a Python script that uses the MClientAPI python module, you can find into the SDK folder examples on how to submit a job. Attaching an environment is like with mrtool, you just add an environmental_variables attribute

So pick the way you think is more comfortable for you and let us know how it goes and if you have further questions

Have a nice day!

Leonardo Bernardini

7th May 2019 at 9:13 am #17491

So you’re problem is only with Maya, I see also that you’re using Maya 2018 and honestly this is not certified with Muster 8.6.1 that’s a very old version.
By the way, if a CMD prompt works, try to stop the Muster render client service, and start the renderclient.exe manually in console mode, then re-try to send the job. If it works in this way, the problem is with your service configuration.

Leonardo Bernardini

7th May 2019 at 9:11 am #17490

Sure, state if it is enabled or disabled, like you can have an attribute filled in console but actually disabled.
The subs flags tells Muster if the attribute should pass the substitution paths engine (i.e. you’re submitting from a different platform than the Dispatcher or in general you’re submitting a path/file path)

Leonardo Bernardini

3rd May 2019 at 8:10 pm #17487

Hi Bernie,

First of all, what Muster version are you using ? you posted into the Muster 8 forum , so maybe I’ll move this topic later.
Then, you have to check the following things:

1) An user is assigned to the Renderclient service, if it’s running in LOCAL SYSTEM ACCOUNT, Maya batch and network connections may not work

2) Start a simple scene first, without using drive mappings, neither pre-render mel scripts and from a network location

3) Once you get a successfull render, configure the drive mappings inside the Renderclient preferences (from console, right click on a node and then select Configure , then locate the Drive mappings sections), and then restart (or soft-restart from the console context menu) the render client

A stuck maya batch 99% of case is due to a wrong user configuration, so I suppose you have not assigned an user to the service.

Let me know if this helps you

Leonardo Bernardini

3rd May 2019 at 10:52 am #17485

Additional note, in the command line you launch, you can use Python inline substitution as explained in the manual, if you need to pass it any job parameter. For a real life example, the must useful thing it to launch the Muster python interpreter (instead of the system one) that’s the mpython executable and pass the job ID to a script. I.e /usr/local/muster9/mpython myscript.py <%=job.getJobId()%> . In that way, you can then import the muster API inside myscript.py, reconnect to the Dispatcher and get the full MJob object back , and perform all your stuff.
Off course this is for a situation where you need to extract all the jobs parameters, if you need just a few parameters, you can just pass the parameters you need directly to the command line and use your python interpreter.

Viewing 15 posts - 76 through 90 (of 492 total)

← 1 2 3 … 5 6 7 … 31 32 33 →