Python – Dynamically loads modules in Python 3 at runtime

What is the recommended way to lazy load files in Python 3?

I built this function by copying from Python 2 code

def get_command(self, ctx, cmd_name):
        ns = {}
        fn = os.path.join(cmd_folder, 'cmd_{}.py'.format(cmd_name))

with open(fn) as f:
            code = compile(f.read(), str(fn), 'exec')
            eval(code, ns, ns)

return ns['cli']

But I’m not sure if that’s the right approach. It uses compile and eval

Edit

After using import like this:

def get_command(self, ctx, cmd_name):
    cmd_mod = import_module('{}.cmd_{}'.format(cmd_folder, cmd_name))

Here is the result:

  File "aws_iam_cli/cli.py", line 23, in get_command
    cmd_mod = import_module('{}.cmd_{}'.format(cmd_folder, cmd_name))
  File "/Users/salvatoremazzarino/awsiam/venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named '/Users/salvatoremazzarino/awsiam/aws_iam_cli/commands'

Edit #2:

def get_command(self, ctx, cmd_name):
        mod = import_module('aws_iam_cli.commands.cmd_{}'
                            .format(cmd_name))
        return mod.cli

and error:

  File "/Users/salvatoremazzarino/awsiam/venv/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'aws_iam_cli'

This is the directory tree:

├── aws_iam_cli
│   ├── __init__.py
│   ├── cli.py
│   ├── commands
│   │   ├── __init__.py
│   │   ├── __pycache__
│   │   └── cmd_dump.py
│   └── provider
│       ├── __init__.py
│       ├── policy.py
│       └── role.py

Internal command I have a module that I call inside the module:

from aws_iam_cli.provider.role import fetch_roles

You can use importlib.import_module to dynamically load modules at runtime. For example:

from importlib import import_module

# load module
pckg_name = 'commands'  # I assume all your commands are in this pckg
cmd_mod = import_module('{}.cmd_{}'.format(pckg_name, cmd_name))

# run command (I'm assuming the module contains a function with the 'cmd_name')
cmd_mod.cmd_name()

I think this approach will be cleaner, safer, and should work with what you’re doing.