Python: How best to handle development libraries and applications that use it at the same time?

Example of the current project directory structure:

myproject/
  |
  +-- mylibrary/
  |     |
  |     +-- __init__.py
  |     |
  |     +-- (code files)
  |
  +-- webapi/
  |     |
  |     +-- __init__.py       <-- contains Flask API code and blueprints, using the mylibrary code
  |     |
  |     +-- object/
  |           |
  |           +-- __init__.py <-- Flask API code for "object"s, imported via blueprints
  |
  +-- cli/
  |     |
  |     +-- __init__.py       <-- argparse, code to use the mylibrary code from the CLI, etc.
  |
  +-- gui_app/
        |
        +-- __init__.py       <-- the start of a GUI application, using the mylibrary code

I’m the only developer and am working on all of the above parts of the application in parallel.

My question:

• How do I best handle writing import statements in three projects (WebAPI, CLI, and gui_app) to import code from the MyLibrary module? I see three options:
  • USE PYTHONPATH OR SYS.PATH TO ADD : TO THE PATH OF EACH APPLICATION THAT USES THE LIBRARY. This works for __init__.py files, but I’m not sure how to do this from webapi/object/__init__.py without adding it: /.. to sys.path. In addition, this makes the application more difficult to distribute later.
  • Refactor the application so that each “user” of the library is a submodule of the library; Then I can execute from: import mylibrary. This is a bad idea because it basically makes the entire mylibrary module “monolithic”.
  • Every time I have a chance to access mylibrary, install it into my site-packages. Then simply develop the other three applications using Import MyLibrary. However, this is very painful because I am developing all four parts of the application in parallel.
• I would plan to distribute mylibrary myself so that it can be made by e.g. pips. You can then install WebAPI or gui_app separately. (Maybe the CLI could be merged into mylibrary so that if it runs as a module, it will show the CLI, but still won’t solve the problem with the other two applications.)

This is the first time I’ve written a codebase, writing multiple applications that will use that library at the same time. What is the most “Pythonic” or, more accurately, the most secure and least error-prone implementation?

I think you complicate things.

Your ultimate goal is that both libraries and applications are distributed separately, installed by pip. To achieve this, they only need to be a standalone package with its own separate setup.py, and the application includes the library in its “requirements.txt”.

And, once you do, it can also solve your development problems. Create a virtual environment and install the library pip into that environment as you develop your application.

No need for manual symbolic links or copying anything anywhere. Or write your tests in such a way that they can use both installed libraries and libraries with relative paths. Or anything else. Don’t mess with sys.path; The library is in venv’s site package, which is already in sys.path. Getting all of this working is what a virtual environment is all about.

You don’t even need anything complicated in the readme. Anyone who installs the application using pip (either system-wide or in venv) automatically gets the library. People working in the application can treat the library like any other dependency. (They should already know how to create a venv and install requirements.txt into it.) The only real problem is that people who work in libraries aren’t interested in the app, but use it as test code for libraries — but you can make it unnecessary by including enough test code in the library itself (which you might want to do anyway).

If that doesn’t work for you, you can look at building a versioned child package from a separate library package, how requests handles urllib3, or bs4 using Unicodedamnit. But from your description, I don’t see any indication that you need it.

I think your problem might be that you think pip can only install distributed packages from PyPI or other pip repositories. In fact, it’s much more flexible than that. As the User Guide says

pip supports installing from PyPI, version control, local projects, and directly from distribution files.

If you look at the reference to pip install, it can take any of the following forms:

pip [options] <requirement specifier> [package-index-options] ...
pip [options] -r <requirements file> [package-index-options] ...
pip [options] [-e] <vcs project url> ...
pip [options] [-e] <local project path> ...
pip [options] <archive url/path> ...

If you’re wondering how it works, simplify it a bit, and all the variations (unless you’re dealing with pre-made wheels, which aren’t relevant here) boil down to (just slightly too simple) downloading the source code/checkout/unzipping/etc. Somewhere and execute pip install., cd goes somewhere, and then pip installs there does.

So, if you want to install the current working tree of your library, you can do this:

pip install /path/to/lib

Or, more commonly, you’re already in /path/to/lib, so:

pip install .

You may want to include the -e flag for development mode installations, or override version checking (or just —force-reinstall), or specify a fake (monotonically incrementing) version number with an #egg, or ignore requirements or constraints, or whatever, but all of these options work from a local path (or git). repo or branch or changelist) is like installing from PyPI.

Even for very complex things, pip handles it well. Want to switch back to what’s on master to compare how your current changes affect things? pip install git+file:/path/to/lib@master。 Want to give someone a custom branch so they can test against it? Push the branch, he can execute pip install git+https://github.com/fdmillion/liblibrary@someguytest. You may never need these things, but pip already has almost everything you can think of.