Access the Muster python API from inside Maya

Tagged: 

Viewing 11 posts - 1 through 11 (of 11 total)
  • 4th June 2019 at 2:53 pm #17682

    I’m trying to import MClientAPI inside Maya 2017, python 2.7.11, but I can’t seem to get it working.
    How would I go about loading the muster API from inside maya, or nuke for that matter?

    I followed you docs and added:
    MUSTER9=C:\program files\Virtual Vertex\Muster 9
    PATH=%MUSTER9%;%PATH%

    If I just append ‘C:/Program Files/Virtual Vertex/Muster 9’ to PATH like above I get:
    ImportError: Module use of python33.dll conflicts with this version of Python.

    If I then pre-append ‘C:/Program Files/Virtual Vertex/Muster 9/sdk/libs/win64/python27’ to PATH i instead get:
    # Error: DLL load failed: The specified module could not be found.
    # Traceback (most recent call last):
    # File “<maya console>”, line 1, in <module>
    # File “C:/Program Files/Virtual Vertex/Muster 9/sdk/libs/win64/python27\MClientAPI.py”, line 24, in <module>
    # _MClientAPI = swig_import_helper()
    # File “C:/Program Files/Virtual Vertex/Muster 9/sdk/libs/win64/python27\MClientAPI.py”, line 23, in swig_import_helper
    # return importlib.import_module(‘_MClientAPI’)
    # File “C:\Program Files\Autodesk\Maya2017\bin\python27.zip\importlib\__init__.py”, line 37, in import_module
    # __import__(name)
    # ImportError: DLL load failed: The specified module could not be found. #

    Any help would be appreciated as I want to try to get away from the old school mrtools approach.

    4th June 2019 at 3:43 pm #17685

    Hi Alex,

    Mixing Python interpreters is always a bad idea. What happens here is that when you bring up inside the Maya Python 2.7.11 the MClientAPI module, it in turns bring ups its own 3.3 interpreter for its internal stuff that deals with the template engine. You should try to re-define also PYTHONHOME and PYTHONPATH pointing to the Muster installation directory. By the way, embedding the Muster API inside Maya/Nuke may be tedious also to maintain. Have you considered using the REST/APIs that’s the way we are going internally for new connectors, and basically let you do any thing you can do with the MClientAPI and in an easier way ? Or is there a particular function in the MClientAPI module you need, so I can suggest you a workaround ?

    4th June 2019 at 4:16 pm #17687

    It’s totally understandable that mixing environments is a bad idea and there’s not really a reason I want to use MClientAPI.
    I’m basically trying to submit a job from our own python submitter and in the past we used mrtool but since you are moving away from that in the future I wanted to “future proof” our submitter now that I’m rewriting it.

    I had a quick look at the RESTful HTTP API but can’t see any examples of how I would use it in combination with our python submitter.
    Maybe I’m just missing where the docs or examples are for something like this or it’s just not been written yet.

    I’m guessing I’m going to need some sort of http module for python in order to interact with the api but don’t know where to start so any help that can point me in the right direction would be awesome.

    4th June 2019 at 4:27 pm #17688

    Sure,

    First of all the docs are at https://www.vvertex.com/wiki900/doku.php?id=muster:9.0:restful_http_api

    Our Houdini-PDG integration script uses the REST APIs to submit a job through Muster, it is into the latest Muster distribution.
    By the way, to have a fine control of what you’re sending to Muster, step 1 is to open the Muster web interface, open the developers tools in Chrome and inspect the POST call that’s sent from the web interface when you send a job, in that way you can debug the exact JSON required by your jobs/template submissions.

    Then, as a general workflow:

    1) In Python 2.x you can use urllib2 to make an http request, it’s a standard Python library and Maya carries it.
    2) You also need to deal with JSON , so importing json is a good thing. Those are the bootstrap imports from our PDG submitter:

    import os
    import sys
    import shutil
    import logging
    import ssl
    import json
    import re
    import threading
    import shlex
    import platform
    
    try:
        _create_unverified_https_context = ssl._create_unverified_context
    except AttributeError:
        # Legacy Python that doesn't verify HTTPS certificates by default
        pass
    else:
        # Handle target environment that doesn't support HTTPS verification
        ssl._create_default_https_context = _create_unverified_https_context
    try:
        import urllib2 as urlreq # Python 2.x
    except:
        import urllib.request as urlreq # Python 3.x
    
    import urllib

    3) Before each submission, you need to obtain an access token from the HTTP APIs, this can be done called /api/login/username=admin&password=password , the response is pure JSON and into the ResponseData , you need to grab the authToken. Little example:

    
    url = self.getBaseDispatcherUrl() + "/api/login?" + urllib.urlencode({'username':dispatcher_login,'password':dispatcher_password})
    req = urlreq.Request(url)
    req.add_header('User-Agent', 'Houdini (PDG)')
    contents = urlreq.urlopen(req,context=context).read()
    res = json.loads(contents)
    if 'ResponseStatus' in res:
        if 'Code' in res['ResponseStatus']:
            retCode = int(res["ResponseStatus"]["Code"])
            if retCode == 200:
                if 'ResponseData' in res:
                    if 'authToken' in res["ResponseData"]:
                        self.authToken = res["ResponseData"]["authToken"]
                    else:
                         logger.error("Authentication failed:{}:{}".format(retCode,res["ResponseStatus"]["Description"]))

    Once you have an access token, you can submit . Also remember to logout to deallocate your session, or it will expire automatically after 20 minutes, but if you’re making massive submissions, logout is good

    4) Submit the job, again this is a sample from PDG, you need to fine tune your job attributes as well as the parameters you’re sending:

     url = self.getBaseDispatcherUrl() + "/api/queue/actions?" + urllib.urlencode({'name':'submit','authToken':self.authToken})
                requestData = "{\n\"RequestData\":\n{\n\"platform\" : " + str(submission_platform) + ",\n\"job\":\n{"
                requestData += "\"jobName\":{},\n".format(json.dumps(job_name))
                requestData += "\"templateId\":{},\n".format(dispatcher_pdg_template_id)
                if len(dispatcher_pdg_template_version):
                    requestData += "\"templateVersion\":{},\n".format(json.dumps(dispatcher_pdg_template_version))
    
                requestData += "\"priority\":{},\n".format(muster_job_priority)
                if muster_job_emergency == "on": requestData += "\"emergencyQueue\":1,\n"
                included_pools = filter(None, muster_job_pool.split(','))
                excluded_pools = filter(None, muster_job_ex_pool.split(','))
                encoded_included_pools = []
                encoded_excluded_pools = []
                for x in included_pools:
                    encoded_included_pools.append(json.dumps(x))
                for x in excluded_pools:
                    encoded_excluded_pools.append(json.dumps(x))
    
                if len(encoded_included_pools):
                        requestData += "\"includedPools\":[" + ','.join(encoded_included_pools) + "],\n"
                if len(encoded_excluded_pools):
                        requestData += "\"excludedPools\":[" + ','.join(encoded_excluded_pools) + "],\n"
              
                if self.parent_id > 0: requestData += "\"parentId\" : " + str(self.parent_id) + ",\n"
                requestData += "\"attributes\":{"
                requestData += "\"PDGEXECUTABLE\" : { \"value\" : " + json.dumps(executable) + " , \"state\" : true, \"subst\" : true }"
                requestData += ",\"PDGARGUMENTS\" : { \"value\" : " + json.dumps(arguments) + " , \"state\" : true, \"subst\" : true }"
                requestData += ",\"memo\" : { \"value\" : " + json.dumps(muster_job_notes) + " , \"state\" : true, \"subst\" : false }"
       
                requestData += "}\n"
                requestData += "}\n}\n}\n"
                req = urlreq.Request(url,requestData)
                req.add_header('Content-Type', 'application/json')
                req.add_header('User-Agent', 'Houdini (PDG)')
                contents = urlreq.urlopen(req,context=context).read()
                res = json.loads(contents)
                if 'ResponseStatus' in res:
                    if 'Code' in res['ResponseStatus']:
                        retCode = int(res["ResponseStatus"]["Code"])
                        description = res["ResponseStatus"]["Description"]
                        if retCode == 200:
                            job_id = res["ResponseStatus"]["objectId"]
                            self.jobs_lock.acquire()
                            self.active_jobs[job_id] = { 'name' : item_name , 'index' : item_id }
                            self.jobs_lock.release()
                            logger.debug('Scheduling: {} {} successfully'.format(node_name, item_name))
                            work_item.data.setInt("muster_jobid", job_id, 0)
                            return scheduleResult.Succeeded
                        else:
                            logger.error("Scheduling failed: {}:{}".format(retCode,description))
                    else:
                         logger.error("Scheduling failed: Invalid response from server")
                else:
                    logger.error("Scheduling failed: Invalid response from server")

    Hope this points you to the right direction. Cheers !

    4th June 2019 at 5:59 pm #17691

    This is awesome. Thanks.
    Hopefully one final question before I jump into the deep.

    Is there a comprehensive list somewhere of all attribute names for the REST API?
    I managed to find a few of them scrolling through the different examples in the doc but I couldn’t find a complete list anywhere
    and there are a few flags I’m using in mrtool that I don’t know what the matching REST names would be.
    Or am I just dumb and should be using the same names as mrtool, unless there’s a specific “MAPS TO” in the docs.
    For example, what would be the correct name and implementation of the following flags?
    -oct, -recursion, -ecerrtype, -logerrtype, -framesmask

    4th June 2019 at 6:01 pm #17692

    No, names are different, that’s why I suggested you to send a job through the Muster web interface with Google Developer Tools open in Chrome, and enable all the attributes. This will send a job with everything enabled, and you can have a base reference point for your JSON.
    If you do not have confidence with Chrome and developer tools, I can provide you a base json with everything enabled.

    4th June 2019 at 6:14 pm #17693

    A base JSON would be great.
    I tried the chrome dev tools but I’m not really experienced enough with it to know how to get any sort of print out of the submission command.

    4th June 2019 at 6:24 pm #17694

    This is a sample job JSON with all the attributes/overrides enabled, off-course you should use just the things you need:

    
    {
        "jobName": "Untitled",
        "templateId": 1,
        "templateVersion": "",
        "parentId": -1,
        "dependIds": [
            "100"
        ],
        "dependMode": 0,
        "dependLinkMode": 0,
        "priority": 1,
        "project": "job-project",
        "department": "job-department",
        "camera": "job-camera",
        "shot": "job-shot",
        "sequence": "job-sequence",
        "includedPools": [],
        "excludedPools": [],
        "owner": "admin",
        "startOn": 12,
        "pauseOn": 0,
        "resumeOn": 0,
        "requeued": 0,
        "packetSize": 4,
        "chunksPriority": 0,
        "chunksInterleave": 2,
        "progress": 0,
        "emergencyQueue": false,
        "packetType": 1,
        "maximumInstances": 0,
        "assignedInstances": 0,
        "exitCodesErrorCheckType": 0,
        "logsErrorCheckType": 0,
        "overrideMailNotificationsAtJobLevel": true,
        "mailNotificationsAtJobLevelType": 1,
        "overrideMailNotificationsAtChunkLevel": true,
        "mailNotificationsAtChunkLevelType": 0,
        "overrideNotificatorNotificationsAtJobLevel": true,
        "notificatorNotificationsAtJobLevelType": 1,
        "overrideNotificatorNotificationsAtChunkLevel": true,
        "notificatorNotificationsAtChunkLevelType": 1,
        "overrideMobileNotificationsAtJobLevel": true,
        "mobileNotificationsAtJobLevelType": 1,
        "overrideMobileNotificationsAtChunkLevel": true,
        "mobileNotificationsAtChunkLevelType": 1,
        "overrideStartMailNotificationsAtJobLevel": true,
        "startMailNotificationsAtJobLevelType": 1,
        "overrideStartMailNotificationsAtChunkLevel": true,
        "startMailNotificationsAtChunkLevelType": 0,
        "overrideStartNotificatorNotificationsAtJobLevel": true,
        "startNotificatorNotificationsAtJobLevelType": 0,
        "overrideStartNotificatorNotificationsAtChunkLevel": true,
        "startNotificatorNotificationsAtChunkLevelType": 1,
        "overrideStartMobileNotificationsAtJobLevel": true,
        "startMobileNotificationsAtJobLevelType": 1,
        "overrideStartMobileNotificationsAtChunkLevel": true,
        "startMobileNotificationsAtChunkLevelType": 1,
        "overrideChunksTimeout": true,
        "chunksTimeoutValue": 320,
        "overrideValidExitCodes": true,
        "overrideWarningExitCodes": true,
        "overrideErrorExitCodes": true,
        "overrideValidExitCodesValue": "1",
        "overrideWarningExitCodesValue": "2",
        "overrideErrorExitCodesValue": "3",
        "overrideMinimumThreads": true,
        "overrideMinimumCores": true,
        "overrideMinimumPhysical": true,
        "overrideMinimumSpeed": true,
        "overrideMinimumRam": true,
        "overrideMinimumDiskSpace": true,
        "overrideMaximumChunksRequeue": true,
        "overrideMaximumChunksRequeueValue": 5,
        "overrideMinimumThreadsValue": 13,
        "overrideMinimumCoresValue": 14,
        "overrideMinimumPhysicalValue": 15,
        "overrideMinimumSpeedValue": 16,
        "overrideMinimumRamValue": 17,
        "overrideMinimumDiskSpaceValue": 18,
        "logsParsingRules": "",
        "updateTime": 1559665190,
        "attributes": {
            "ABORTRENDER": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "ADD_FLAGS": {
                "value": "-addflag",
                "state": true,
                "subst": true
            },
            "MAYADIGITS": {
                "value": "1",
                "state": true,
                "subst": false
            },
            "by_frame": {
                "value": "1",
                "state": true,
                "subst": false
            },
            "confirmed_mask": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "end_frame": {
                "value": "10",
                "state": true,
                "subst": false
            },
            "environmental_variables": {
                "value": "ENV=1",
                "state": true,
                "subst": false
            },
            "fc_check_file": {
                "value": "1",
                "state": true,
                "subst": false
            },
            "fc_check_image_dimensions": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "fc_file_high": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "fc_file_low": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "fc_image_height": {
                "value": "1080",
                "state": true,
                "subst": false
            },
            "fc_image_width": {
                "value": "1920",
                "state": true,
                "subst": false
            },
            "fc_open_image": {
                "value": "1",
                "state": true,
                "subst": false
            },
            "fc_recursion": {
                "value": "1",
                "state": true,
                "subst": false
            },
            "fc_render_layer_mode": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "frame_check": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "frames_mask": {
                "value": "QSBmcmFtZSBtYXNrCi9wYXRoL3RvL2ZyYW1lL21hc2sK",
                "state": true,
                "subst": true
            },
            "frames_range": {
                "value": "1-10",
                "state": true,
                "subst": false
            },
            "job_file": {
                "value": "/path/to/maya/scene.mb",
                "state": true,
                "subst": true
            },
            "job_project": {
                "value": "/path/to/maya",
                "state": true,
                "subst": true
            },
            "memo": {
                "value": "job-notes",
                "state": true,
                "subst": false
            },
            "movie_assembler": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "movie_assembler_framerate": {
                "value": "30",
                "state": true,
                "subst": false
            },
            "movie_assembler_input_flags": {
                "value": "",
                "state": false,
                "subst": false
            },
            "movie_assembler_output": {
                "value": "",
                "state": true,
                "subst": true
            },
            "movie_assembler_output_flags": {
                "value": "",
                "state": false,
                "subst": false
            },
            "movie_assembler_template": {
                "value": "48",
                "state": true,
                "subst": false
            },
            "output_folder": {
                "value": "/path/to/frames/destination",
                "state": true,
                "subst": true
            },
            "overridestatus": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "post_chunk_action": {
                "value": "/post/chunk/action.cmd",
                "state": true,
                "subst": true
            },
            "post_chunk_action_check_retcode": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "post_chunk_action_timeout": {
                "value": "20",
                "state": true,
                "subst": false
            },
            "post_job_action": {
                "value": "/post/job/action.cmd",
                "state": true,
                "subst": true
            },
            "post_job_action_check_retcode": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "post_job_action_timeout": {
                "value": "20",
                "state": true,
                "subst": false
            },
            "pre_chunk_action": {
                "value": "/pre/chunk/action.cmd",
                "state": true,
                "subst": true
            },
            "pre_chunk_action_check_retcode": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "pre_chunk_action_timeout": {
                "value": "20",
                "state": true,
                "subst": false
            },
            "pre_job_action": {
                "value": "/pre/job/action.cmd",
                "state": true,
                "subst": true
            },
            "pre_job_action_check_retcode": {
                "value": "0",
                "state": true,
                "subst": false
            },
            "pre_job_action_timeout": {
                "value": "20",
                "state": true,
                "subst": false
            },
            "start_frame": {
                "value": "1",
                "state": true,
                "subst": false
            }
        }
    }
    
    4th June 2019 at 6:27 pm #17695

    Also to complete the explanation, there’s another way to submit a job using the MClientAPI , but instead of importing the module inside the running Python environment, you would spawn MPython.exe (inside the Muster installation folder) that comes preconfigured, and pass your custom script that uses the MClientAPI functions as well as passing some parameters grabbed from Maya itself. If you need the power of Python to pre-flight your job, this may be another option you may consider.

    4th June 2019 at 7:05 pm #17697

    Great, thanks.

    I’ve manage to get all attributes working except frames_mask. It enables the attribute but it stays “No layers masks”.

    I’m using the following:
    “frames_mask”:
    {
    “value”: “layer path”,
    “state”: true,
    “subst”: false
    }

    What’s the formatting for this flag?

    5th June 2019 at 12:00 pm #17714

    “frames_mask”: {
    “value”: “QSBmcmFtZSBtYXNrCi9wYXRoL3RvL2ZyYW1lL21hc2sK”,
    “state”: true,
    “subst”: true
    },

    the content must be BASE64 encoded of basically this:

    1) Layer name + carriage return
    2) Layer file mask path + carriage return

    It can contain multiple layers just using multiple lines.

Viewing 11 posts - 1 through 11 (of 11 total)

You must be logged in to reply to this topic.