Python – When I try to install caffe on Windows 10, the script can’t find the python library and numpy

When I try to install caffe on Windows 10, the script can’t find the python library and numpy… here is a solution to the problem.

When I try to install caffe on Windows 10, the script can’t find the python library and numpy

I’ve been using this resource to install caffe on my Windows 10 system.

I started by installing the necessary dependencies for Python via the command prompt:

C:\Users\MYNAME>conda config --add channels conda-forge
C:\Users\MYNAME>conda config --add channels willyd
C:\Users\MYNAME>conda install --yes cmake ninja numpy scipy protobuf==3.1.0 six scikit-image pyyaml pydotplus graphviz

Then I created an empty directory called caffe in my C:\ directory. Then I went into that directory through the command prompt and typed the following command:

C:\caffe\caffe>git clone https://github.com/BVLC/caffe.git
C:\caffe\caffe>cd caffe

Then I opened the file build_win.cmd in the /scripts directory and edited something. The edited file looks like this:

@echo off
@setlocal EnableDelayedExpansion

:: Default values
if DEFINED APPVEYOR (
    echo Setting Appveyor defaults
    if NOT DEFINED MSVC_VERSION set MSVC_VERSION=14
    if NOT DEFINED WITH_NINJA set WITH_NINJA=0
    if NOT DEFINED CPU_ONLY set CPU_ONLY=1
    if NOT DEFINED CUDA_ARCH_NAME set CUDA_ARCH_NAME=Auto
    if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release
    if NOT DEFINED USE_NCCL set USE_NCCL=0
    if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0
    if NOT DEFINED PYTHON_VERSION set PYTHON_VERSION=3
    if NOT DEFINED BUILD_PYTHON set BUILD_PYTHON=1
    if NOT DEFINED BUILD_PYTHON_LAYER set BUILD_PYTHON_LAYER=1
    if NOT DEFINED BUILD_MATLAB set BUILD_MATLAB=0
    if NOT DEFINED PYTHON_EXE set PYTHON_EXE=python
    if NOT DEFINED RUN_TESTS set RUN_TESTS=1
    if NOT DEFINED RUN_LINT set RUN_LINT=1
    if NOT DEFINED RUN_INSTALL set RUN_INSTALL=1

:: Set python 2.7 with conda as the default python
if ! PYTHON_VERSION! EQU 2 (
    **set CONDA_ROOT=C:\Users\MYNAME\Anaconda3**
)
:: Set python 3.5 with conda as the default python
if ! PYTHON_VERSION! EQU 3 (
    **set CONDA_ROOT=C:\Users\MYNAME\Anaconda3**
)
**set PATH=C:\Users\MYNAME\Anaconda3; C:\Users\MYNAME\Anaconda3\Scripts; C:\Users\MYNAME\Anaconda3\Library\bin;! PATH!**

:: Check that we have the right python version
! PYTHON_EXE! --version
:: Add the required channels
conda config --add channels conda-forge
conda config --add channels willyd
:: Update conda
conda update conda -y
:: Download other required packages
conda install --yes cmake ninja numpy scipy protobuf==3.1.0 six scikit-image pyyaml pydotplus graphviz

if ERRORLEVEL 1  (
  echo ERROR: Conda update or install failed
  exit /b 1
)

:: Install cuda and disable tests if needed
if ! WITH_CUDA! == 1 (
    call %~dp0\appveyor\appveyor_install_cuda.cmd
    set CPU_ONLY=0
    set RUN_TESTS=0
    set USE_NCCL=1
) else (
    set CPU_ONLY=1
)

:: Disable the tests in debug config
if "%CMAKE_CONFIG%" == "Debug" (
    echo Disabling tests on appveyor with config == %CMAKE_CONFIG%
    set RUN_TESTS=0
)

:: Disable linting with python 3 until we find why the script fails
if ! PYTHON_VERSION! EQU 3 (
    set RUN_LINT=0
)

) else (
    :: Change the settings here to match your setup
    :: Change MSVC_VERSION to 12 to use VS 2013
    if NOT DEFINED MSVC_VERSION set MSVC_VERSION=14
    :: Change to 1 to use Ninja generator (builds much faster)
    if NOT DEFINED WITH_NINJA set WITH_NINJA=0
    :: Change to 1 to build caffe without CUDA support
    if NOT DEFINED CPU_ONLY set CPU_ONLY=1
    :: Change to generate CUDA code for one of the following GPU architectures
    :: [Fermi  Kepler  Maxwell  Pascal  All]
    if NOT DEFINED CUDA_ARCH_NAME set CUDA_ARCH_NAME=Auto
    :: Change to Debug to build Debug. This is only relevant for the Ninja generator the Visual Studio generator will generate both Debug and Release configs
    if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release
    :: Set to 1 to use NCCL
    if NOT DEFINED USE_NCCL set USE_NCCL=0
    :: Change to 1 to build a caffe.dll
    if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0
    :: Change to 3 if using python 3.5 (only 2.7 and 3.5 are supported)
    if NOT DEFINED PYTHON_VERSION set PYTHON_VERSION=3
    :: Change these options for your needs.
    if NOT DEFINED BUILD_PYTHON set BUILD_PYTHON=1
    if NOT DEFINED BUILD_PYTHON_LAYER set BUILD_PYTHON_LAYER=1
    if NOT DEFINED BUILD_MATLAB set BUILD_MATLAB=0
    :: If python is on your path leave this alone
    if NOT DEFINED PYTHON_EXE set PYTHON_EXE=python
    :: Run the tests
    if NOT DEFINED RUN_TESTS set RUN_TESTS=0
    :: Run lint
    if NOT DEFINED RUN_LINT set RUN_LINT=0
    :: Build the install target
    if NOT DEFINED RUN_INSTALL set RUN_INSTALL=0
)

:: Set the appropriate CMake generator
:: Use the exclamation mark ! below to delay the
:: expansion of CMAKE_GENERATOR
if %WITH_NINJA% EQU 0 (
    if "%MSVC_VERSION%"=="14" (
        set CMAKE_GENERATOR=Visual Studio 14 2015 Win64
    )
    if "%MSVC_VERSION%"=="12" (
        set CMAKE_GENERATOR=Visual Studio 12 2013 Win64
    )
    if "! CMAKE_GENERATOR!" =="" (
        echo ERROR: Unsupported MSVC version
        exit /B 1
    )
) else (
    set CMAKE_GENERATOR=Ninja
)

echo INFO: ============================================================
echo INFO: Summary:
echo INFO: ============================================================
echo INFO: MSVC_VERSION               = ! MSVC_VERSION!
echo INFO: WITH_NINJA                 = ! WITH_NINJA!
echo INFO: CMAKE_GENERATOR            = "! CMAKE_GENERATOR!"
echo INFO: CPU_ONLY                   = ! CPU_ONLY!
echo INFO: CUDA_ARCH_NAME             = ! CUDA_ARCH_NAME!
echo INFO: CMAKE_CONFIG               = ! CMAKE_CONFIG!
echo INFO: USE_NCCL                   = ! USE_NCCL!
echo INFO: CMAKE_BUILD_SHARED_LIBS    = ! CMAKE_BUILD_SHARED_LIBS!
echo INFO: PYTHON_VERSION             = ! PYTHON_VERSION!
echo INFO: BUILD_PYTHON               = ! BUILD_PYTHON!
echo INFO: BUILD_PYTHON_LAYER         = ! BUILD_PYTHON_LAYER!
echo INFO: BUILD_MATLAB               = ! BUILD_MATLAB!
echo INFO: PYTHON_EXE                 = "! PYTHON_EXE!"
echo INFO: RUN_TESTS                  = ! RUN_TESTS!
echo INFO: RUN_LINT                   = ! RUN_LINT!
echo INFO: RUN_INSTALL                = ! RUN_INSTALL!
echo INFO: ============================================================

:: Build and exectute the tests
:: Do not run the tests with shared library
if ! RUN_TESTS! EQU 1 (
    if %CMAKE_BUILD_SHARED_LIBS% EQU 1 (
        echo WARNING: Disabling tests with shared library build
        set RUN_TESTS=0
    )
)

if NOT EXIST build mkdir build
pushd build

:: Setup the environement for VS x64
set batch_file=! VS%MSVC_VERSION%0COMNTOOLS!.. \.. \VC\vcvarsall.bat
call "%batch_file%" amd64

:: Configure using cmake and using the caffe-builder dependencies
:: Add -DCUDNN_ROOT=C:/Projects/caffe/cudnn-8.0-windows10-x64-v5.1/cuda ^
:: below to use cuDNN
cmake -G"! CMAKE_GENERATOR!" ^
      -DBLAS=Open ^
      -DCMAKE_BUILD_TYPE:STRING=%CMAKE_CONFIG% ^
      -DBUILD_SHARED_LIBS:BOOL=%CMAKE_BUILD_SHARED_LIBS% ^
      -DBUILD_python:BOOL=%BUILD_PYTHON% ^
      -DBUILD_python_layer:BOOL=%BUILD_PYTHON_LAYER% ^
      -DBUILD_matlab:BOOL=%BUILD_MATLAB% ^
      -DCPU_ONLY:BOOL=%CPU_ONLY% ^
      -DCOPY_PREREQUISITES:BOOL=1 ^
      -DINSTALL_PREREQUISITES:BOOL=1 ^
      -DUSE_NCCL:BOOL=! USE_NCCL! ^
      -DCUDA_ARCH_NAME:STRING=%CUDA_ARCH_NAME% ^
      "%~dp0\.."

if ERRORLEVEL 1 (
  echo ERROR: Configure failed
  exit /b 1
)

:: Lint
if %RUN_LINT% EQU 1 (
    cmake --build . --target lint  --config %CMAKE_CONFIG%
)

if ERRORLEVEL 1 (
  echo ERROR: Lint failed
  exit /b 1
)

:: Build the library and tools
cmake --build . --config %CMAKE_CONFIG%

if ERRORLEVEL 1 (
  echo ERROR: Build failed
  exit /b 1
)

:: Build and exectute the tests
if ! RUN_TESTS! EQU 1 (
    cmake --build . --target runtest --config %CMAKE_CONFIG%

if ERRORLEVEL 1 (
        echo ERROR: Tests failed
        exit /b 1
    )

if %BUILD_PYTHON% EQU 1 (
        if %BUILD_PYTHON_LAYER% EQU 1 (
            :: Run python tests only in Release build since
            :: the _caffe module is _caffe-d is debug
            if "%CMAKE_CONFIG%"=="Release" (
                :: Run the python tests
                cmake --build . --target pytest

if ERRORLEVEL 1 (
                    echo ERROR: Python tests failed
                    exit /b 1
                )
            )
        )
    )
)

if %RUN_INSTALL% EQU 1 (
    cmake --build . --target install --config %CMAKE_CONFIG%
)

popd
@endlocal

After this, I ran the following command:

C:\caffe\caffe>scripts\build_win.cmd

This gives me the following output:

The system cannot find the drive specified.
The system cannot find the drive specified.
INFO: ============================================================
INFO: Summary:
INFO: ============================================================
INFO: MSVC_VERSION               = 14
INFO: WITH_NINJA                 = 0
INFO: CMAKE_GENERATOR            = "Visual Studio 14 2015 Win64"
INFO: CPU_ONLY                   = 0
INFO: CUDA_ARCH_NAME             = Auto
INFO: CMAKE_CONFIG               = Release
INFO: USE_NCCL                   = 0
INFO: CMAKE_BUILD_SHARED_LIBS    = 0
INFO: PYTHON_VERSION             = 2
INFO: BUILD_PYTHON               = 1
INFO: BUILD_PYTHON_LAYER         = 1
INFO: BUILD_MATLAB               = 0
INFO: PYTHON_EXE                 = "python"
INFO: RUN_TESTS                  = 0
INFO: RUN_LINT                   = 0
INFO: RUN_INSTALL                = 0
INFO: ============================================================
-- The C compiler identification is MSVC 19.0.24215.1
-- The CXX compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler using: Visual Studio 14 2015 Win64
-- Check for working C compiler using: Visual Studio 14 2015 Win64 -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler using: Visual Studio 14 2015 Win64
-- Check for working CXX compiler using: Visual Studio 14 2015 Win64 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found PythonInterp: C:/cygwin64/bin/python2.7.exe (found suitable version "2.7.14", minimum required is "2.7")
-- Downloading prebuilt dependencies to C:/Users/Abdullah Siddiqui/.caffe/dependencies/download/libraries_v140_x64_py27_1.1.0.tar.bz2
-- [download 0% complete]
...
-- Extracting dependencies
-- Looking for pthread.h
-- Looking for pthread.h - not found
-- Found Threads: TRUE
-- Boost version: 1.61.0
-- Found the following Boost libraries:
--   system
--   thread
--   filesystem
--   chrono
--   date_time
--   atomic
-- Found GFlags: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Found gflags  (include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include, library: gflags_shared)
-- Found Glog: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Found glog    (include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include, library: glog)
-- Found Protobuf: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/bin/protoc.exe (found version "3.1.0")
-- Found PROTOBUF Compiler: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/bin/protoc.exe
-- Found LMDB: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Found lmdb    (include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include, library: lmdb)
-- Found LevelDB: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Found LevelDB (include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include, library: leveldb)
-- Found ZLIB: optimized; C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/lib/caffezlib.lib; debug; C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/lib/caffezlibd.lib (found version "1.2.8")
-- Found Snappy: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Found Snappy  (include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include, library: snappy_static;optimized; C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/lib/caffezlib.lib; debug; C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/lib/caffezlibd.lib)
CMake Warning at cmake/Dependencies.cmake:97 (message):
  -- CUDA is not detected by cmake.  Building without it...
Call Stack (most recent call first):
  CMakeLists.txt:80 (include)

-- OpenCV found (C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries)
-- Found OpenBLAS libraries: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/lib/libopenblas.dll.a
-- Found OpenBLAS include: C:/Users/Abdullah Siddiqui/.caffe/dependencies/libraries_v140_x64_py27_1.1.0/libraries/include
-- Could NOT find PythonLibs (missing:  PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS) (Required is at least version "2.7")
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named numpy
-- Could NOT find NumPy (missing:  NUMPY_INCLUDE_DIR NUMPY_VERSION) (Required is at least version "1.7.1")
-- Boost version: 1.61.0
-- Found the following Boost libraries:
--   python
-- Python interface is disabled or not all required dependencies found. Building without it...
-- Found Git: C:/Program Files/Git/cmd/git.exe (found version "2.10.1.windows.1")
--
-- ******************* Caffe Configuration Summary *******************
-- General:
--   Version           :   1.0.0
--   Git               :   1.0-111-g509dae0-dirty
--   System            :   Windows
--   C++ compiler      :   C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/x86_amd64/cl.exe
--   Release CXX flags :   /MD /O2 /Ob2 /D NDEBUG /DWIN32 /D_WINDOWS /W3 /GR /EHsc
--   Debug CXX flags   :   /D_DEBUG /MDd /Zi /Ob0 /Od /RTC1 /DWIN32 /D_WINDOWS /W3 /GR /EHsc
--   Build type        :   Release
--
--   BUILD_SHARED_LIBS :   0
--   BUILD_python      :   1
--   BUILD_matlab      :   0
--   BUILD_docs        :
--   CPU_ONLY          :   0
--   USE_OPENCV        :   ON
--   USE_LEVELDB       :   ON
--   USE_LMDB          :   ON
--   USE_NCCL          :   0
--   ALLOW_LMDB_NOLOCK :   OFF
--
-- Dependencies:
--   BLAS              :   Yes (Open)
--   Boost             :   Yes (ver. 1.61)
--   glog              :   Yes
--   gflags            :   Yes
--   protobuf          :   Yes (ver. 3.1.0)
--   lmdb              :   Yes (ver. 0.9.70)
--   LevelDB           :   Yes (ver. 1.18)
--   Snappy            :   Yes (ver. 1.1.1)
--   OpenCV            :   Yes (ver. 3.1.0)
--   CUDA              :   No
--
-- Install:
--   Install path      :   C:/Users/Abdullah Siddiqui/caffe/build/install
--
-- Configuring done
CMake Error at CMakeLists.txt:143 (add_dependencies):
  The dependency target "pycaffe" of target "pytest" does not exist.

-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

BUILD_python_layer
    CUDA_ARCH_NAME

-- Build files have been written to: C:/Users/Abdullah Siddiqui/caffe/build
ERROR: Configure failed

I’m particularly confused by these statements :

-- Could NOT find PythonLibs (missing:  PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS) (Required is at least version "2.7")
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named numpy
-- Could NOT find NumPy (missing:  NUMPY_INCLUDE_DIR NUMPY_VERSION) (Required is at least version "1.7.1")

How are these files not detectable in my system when build_win.cmd explicitly contains the following line:

set PATH=C:\Users\MYNAME\Anaconda3; C:\Users\ MYNAME\Anaconda3\Scripts; C:\Users\MYNAME\Anaconda3\Library\bin;! PATH!

The related python files also exist in my system.
The .cmd file cannot find PythonLibs in the Anaconda3 directory.

enter image description here

Numpy also exists in a subdirectory of the directory.

enter image description here

I can even run python directly from the command line:

enter image description here

At this point, I don’t know why I’m getting an error about numpy and Python libraries. Let me know if you have any suggestions.

Edit:

After reading this video, I ran the following command in my project directory:

C:\caffe\caffe>python scripts\download_prebuilt_dependencies.py --msvc_version=v140

The output is as follows:

ERROR: Could not find url for MSVC version = v140 and Python version = 3.6.
Available combinations are: [('v120', '2.7'), ('v140', '2.7'), ('v140', '3.5')]

Does this mean I have to switch to Python 3.5 to use this framework?

Edit:

I downloaded python 3.5.5, performed the same steps as above, and got the same error.

I thought more about CristiFati’s suggestion and executed the following command:

C:\caffe\caffe\build>cmake -DPYTHON_EXECUTABLE=C:\Users\Abdullah Siddiqui\Anaconda3\python.exe ..

This gives me the following output:

-- Building for: Visual Studio 14 2015
-- The C compiler identification is MSVC 19.0.24215.1
-- The CXX compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler using: Visual Studio 14 2015
-- Check for working C compiler using: Visual Studio 14 2015 -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler using: Visual Studio 14 2015
-- Check for working CXX compiler using: Visual Studio 14 2015 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Could NOT find PythonInterp: Found unsuitable version "1.4", but required is at least "2.7" (found C:/Users/Abdullah)
CMake Error at cmake/WindowsDownloadPrebuiltDependencies.cmake:40 (message):
  Could not find url for MSVC version = 1900 and Python version = 1.4.
Call Stack (most recent call first):
  CMakeLists.txt:77 (include)

-- Configuring incomplete, errors occurred!
See also "C:/Users/.../projects/caffe/build/CMakeFiles/CMakeOutput.log".

I now understand that the problem is caused by the Python 2.7 executable being used to compile. How do I get my build script to point to Python 3.5?

Edit:

I

ran the cmake command correctly, as shown below, but I now see the new error.

C:\Users\MYNAME\projects\caffe\build>cmake -DPYTHON_EXECUTABLE="C:\Users\MYNAME\Anaconda3\python.exe" ..

It gives me these errors :

-- Could NOT find Glog (missing:  GLOG_LIBRARY)
CMake Error at cmake/ProtoBuf.cmake:6 (find_package):
  Could not find a configuration file for package "Protobuf" that is
  compatible with requested version "".

Solution

cmake has a “standard” way to locate external package installations (via scripts). This applies to Python. When such a package is found, some variables are set, usually:

  • ${PACKAGE}_FOUND=ON
  • ${PACKAGE}_INCLUDE_DIRS (including directories – at compile time).
  • ${PACKAGE}_LIBRARIES (library – used when linking).

When searching for numpy ( [GitHub]: BVLC/caffe – (windows) caffe/cmake/Modules/FindNumPy.cmake) when searching for PythonInterp (standard cmake module) (and use). As shown in the initial output:

-- Found PythonInterp: C:/cygwin64/bin/python2.7.exe (found suitable version "2.7.14", minimum required is "2.7")

Python 2.7 (no numpy module installed) was detected, causing numpy discovery to fail.

Looking at the FindPythonInterp.cmake script, I noticed that it sets (among other things) the ${PYTHON_EXECUTABLE} (cmake) variable.

Setting that variable “manually” to cmake (by passing the cmdline parameter) would be the next logical thing to do (since I don’t know if this alone is enough, I just added a comment):

-DPYTHON_EXECUTABLE="c:\path\to\python3"

Note: Do not mix Cygwin and native Win tools, as you may encounter subtle issues.

Related Problems and Solutions