From 52f7402e1409a6f23f7b8148072e680e841a2f64 Mon Sep 17 00:00:00 2001 From: Pavle Josipovic Date: Thu, 11 Feb 2016 18:03:10 +0100 Subject: [PATCH 01/51] Add support for windows build Necessary changes to enable widndows build: Visual studio project files. Minor code chnages. 3rd party dependencies are resolved through NuGet. --- .gitattributes | 63 ++ .gitignore | 8 + appveyor.yml | 19 + include/caffe/test/test_caffe_main.hpp | 4 +- include/caffe/util/cudnn.hpp | 2 + include/caffe/util/io.hpp | 25 +- python/caffe/test/test_python_layer.py | 4 + python/caffe/test/test_solver.py | 5 +- src/caffe/common.cpp | 9 + src/caffe/layers/bnll_layer.cu | 2 +- src/caffe/test/test_blob.cpp | 2 + src/caffe/test/test_gradient_based_solver.cpp | 15 +- src/caffe/test/test_lrn_layer.cpp | 9 +- src/caffe/util/db_lmdb.cpp | 14 + src/caffe/util/hdf5.cpp | 55 +- src/caffe/util/io.cpp | 9 + tools/caffe.cpp | 6 + windows/Caffe.sln | 87 +++ windows/CommonSettings.props.example | 99 +++ windows/CommonSettings.targets | 11 + windows/caffe/caffe.vcxproj | 118 +++ windows/caffe/packages.config | 18 + .../compute_image_mean/compute_image_mean.vcxproj | 112 +++ windows/compute_image_mean/packages.config | 18 + windows/convert_imageset/convert_imageset.vcxproj | 112 +++ windows/convert_imageset/packages.config | 18 + windows/extract_features/extract_features.vcxproj | 118 +++ windows/extract_features/packages.config | 18 + windows/libcaffe/libcaffe.vcxproj | 383 ++++++++++ windows/libcaffe/libcaffe.vcxproj.filters | 794 +++++++++++++++++++++ windows/libcaffe/packages.config | 14 + windows/nuget.config | 4 + windows/pycaffe/packages.config | 19 + windows/pycaffe/pycaffe.vcxproj | 129 ++++ windows/scripts/BinplaceCudaDependencies.cmd | 22 + windows/scripts/FixGFlagsNaming.cmd | 24 + windows/scripts/ProtoCompile.cmd | 27 + windows/scripts/PythonPostBuild.cmd | 8 + windows/scripts/PythonPreBuild.cmd | 15 + windows/test_all/packages.config | 18 + windows/test_all/test_all.vcxproj | 207 ++++++ windows/test_all/test_all.vcxproj.filters | 232 ++++++ 42 files changed, 2845 insertions(+), 31 deletions(-) create mode 100644 .gitattributes create mode 100644 appveyor.yml create mode 100644 windows/Caffe.sln create mode 100644 windows/CommonSettings.props.example create mode 100644 windows/CommonSettings.targets create mode 100644 windows/caffe/caffe.vcxproj create mode 100644 windows/caffe/packages.config create mode 100644 windows/compute_image_mean/compute_image_mean.vcxproj create mode 100644 windows/compute_image_mean/packages.config create mode 100644 windows/convert_imageset/convert_imageset.vcxproj create mode 100644 windows/convert_imageset/packages.config create mode 100644 windows/extract_features/extract_features.vcxproj create mode 100644 windows/extract_features/packages.config create mode 100644 windows/libcaffe/libcaffe.vcxproj create mode 100644 windows/libcaffe/libcaffe.vcxproj.filters create mode 100644 windows/libcaffe/packages.config create mode 100644 windows/nuget.config create mode 100644 windows/pycaffe/packages.config create mode 100644 windows/pycaffe/pycaffe.vcxproj create mode 100644 windows/scripts/BinplaceCudaDependencies.cmd create mode 100644 windows/scripts/FixGFlagsNaming.cmd create mode 100644 windows/scripts/ProtoCompile.cmd create mode 100644 windows/scripts/PythonPostBuild.cmd create mode 100644 windows/scripts/PythonPreBuild.cmd create mode 100644 windows/test_all/packages.config create mode 100644 windows/test_all/test_all.vcxproj create mode 100644 windows/test_all/test_all.vcxproj.filters diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 00000000000..1ff0c423042 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,63 @@ +############################################################################### +# Set default behavior to automatically normalize line endings. +############################################################################### +* text=auto + +############################################################################### +# Set default behavior for command prompt diff. +# +# This is need for earlier builds of msysgit that does not have it on by +# default for csharp files. +# Note: This is only used by command line +############################################################################### +#*.cs diff=csharp + +############################################################################### +# Set the merge driver for project and solution files +# +# Merging from the command prompt will add diff markers to the files if there +# are conflicts (Merging from VS is not affected by the settings below, in VS +# the diff markers are never inserted). Diff markers may cause the following +# file extensions to fail to load in VS. An alternative would be to treat +# these files as binary and thus will always conflict and require user +# intervention with every merge. To do so, just uncomment the entries below +############################################################################### +#*.sln merge=binary +#*.csproj merge=binary +#*.vbproj merge=binary +#*.vcxproj merge=binary +#*.vcproj merge=binary +#*.dbproj merge=binary +#*.fsproj merge=binary +#*.lsproj merge=binary +#*.wixproj merge=binary +#*.modelproj merge=binary +#*.sqlproj merge=binary +#*.wwaproj merge=binary + +############################################################################### +# behavior for image files +# +# image files are treated as binary by default. +############################################################################### +#*.jpg binary +#*.png binary +#*.gif binary + +############################################################################### +# diff behavior for common document formats +# +# Convert binary document formats to text before diffing them. This feature +# is only available from the command line. Turn it on by uncommenting the +# entries below. +############################################################################### +#*.doc diff=astextplain +#*.DOC diff=astextplain +#*.docx diff=astextplain +#*.DOCX diff=astextplain +#*.dot diff=astextplain +#*.DOT diff=astextplain +#*.pdf diff=astextplain +#*.PDF diff=astextplain +#*.rtf diff=astextplain +#*.RTF diff=astextplain diff --git a/.gitignore b/.gitignore index 53c1fb056bb..15165e7f0db 100644 --- a/.gitignore +++ b/.gitignore @@ -93,3 +93,11 @@ LOCK LOG* CURRENT MANIFEST-* + +#Visual Studio files +*.user +*.suo +*.sdf +*.opensdf +*.pdb +*.props \ No newline at end of file diff --git a/appveyor.yml b/appveyor.yml new file mode 100644 index 00000000000..235cc83dda3 --- /dev/null +++ b/appveyor.yml @@ -0,0 +1,19 @@ +version: 1.0.{build} +clone_folder: c:\projects\caffe +build_script: +- cmd: >- + cd C:\projects\caffe\windows + + copy CommonSettings.props.example CommonSettings.props + + nuget restore Caffe.sln -PackagesDirectory ..\..\NugetPackages -ConfigFile nuget.config + + set PATH=%PATH:nuget=hello% + + msbuild Caffe.sln /m /v:m /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" /p:Configuration=Debug;CpuOnlyBuild=true;UseCuDNN=false + + msbuild Caffe.sln /m /v:m /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" /p:Configuration=Release;CpuOnlyBuild=true;UseCuDNN=false + + cd .. + + Build\x64\Release\test_all.exe --gtest_filter=-*TestTimer* \ No newline at end of file diff --git a/include/caffe/test/test_caffe_main.hpp b/include/caffe/test/test_caffe_main.hpp index fc156091476..12e6d7971ea 100644 --- a/include/caffe/test/test_caffe_main.hpp +++ b/include/caffe/test/test_caffe_main.hpp @@ -10,6 +10,7 @@ #include #include "caffe/common.hpp" +#include "caffe/util/io.hpp" using std::cout; using std::endl; @@ -35,7 +36,8 @@ class MultiDeviceTest : public ::testing::Test { MultiDeviceTest() { Caffe::set_mode(TypeParam::device); } - virtual ~MultiDeviceTest() {} + // Caffe tests may create some temporary files, here we will do the cleanup. + virtual ~MultiDeviceTest() { RemoveCaffeTempDir(); } }; typedef ::testing::Types TestDtypes; diff --git a/include/caffe/util/cudnn.hpp b/include/caffe/util/cudnn.hpp index 8a7e17c6cd4..a1e8ff4d64c 100644 --- a/include/caffe/util/cudnn.hpp +++ b/include/caffe/util/cudnn.hpp @@ -17,6 +17,7 @@ << cudnnGetErrorString(status); \ } while (0) +#if !defined (_MSC_VER) inline const char* cudnnGetErrorString(cudnnStatus_t status) { switch (status) { case CUDNN_STATUS_SUCCESS: @@ -44,6 +45,7 @@ inline const char* cudnnGetErrorString(cudnnStatus_t status) { } return "Unknown cudnn status"; } +#endif namespace caffe { diff --git a/include/caffe/util/io.hpp b/include/caffe/util/io.hpp index 1a599883ca3..6375a4e3be5 100644 --- a/include/caffe/util/io.hpp +++ b/include/caffe/util/io.hpp @@ -23,8 +23,12 @@ using ::boost::filesystem::path; inline void MakeTempDir(string* temp_dirname) { temp_dirname->clear(); - const path& model = - boost::filesystem::temp_directory_path()/"caffe_test.%%%%-%%%%"; + // Place all temp directories under temp_root, to be able to delete all of + // them at once, without knowing their name. + const path& temp_root = + boost::filesystem::temp_directory_path() / "caffe_test"; + boost::filesystem::create_directory(temp_root); + const path& model = temp_root / "%%%%-%%%%"; for ( int i = 0; i < CAFFE_TMP_DIR_RETRIES; i++ ) { const path& dir = boost::filesystem::unique_path(model).string(); bool done = boost::filesystem::create_directory(dir); @@ -37,7 +41,7 @@ inline void MakeTempDir(string* temp_dirname) { } inline void MakeTempFilename(string* temp_filename) { - static path temp_files_subpath; + path temp_files_subpath; static uint64_t next_temp_file = 0; temp_filename->clear(); if ( temp_files_subpath.empty() ) { @@ -49,6 +53,21 @@ inline void MakeTempFilename(string* temp_filename) { (temp_files_subpath/caffe::format_int(next_temp_file++, 9)).string(); } +#ifdef _MSC_VER + +inline void RemoveCaffeTempDir() { + boost::system::error_code err; + boost::filesystem::remove_all( + boost::filesystem::temp_directory_path() / "caffe_test", err); +} + +#else + +inline void RemoveCaffeTempDir() { +} + +#endif + bool ReadProtoFromTextFile(const char* filename, Message* proto); inline bool ReadProtoFromTextFile(const string& filename, Message* proto) { diff --git a/python/caffe/test/test_python_layer.py b/python/caffe/test/test_python_layer.py index e46b7118014..5669451fc68 100644 --- a/python/caffe/test/test_python_layer.py +++ b/python/caffe/test/test_python_layer.py @@ -131,6 +131,10 @@ def test_parameter(self): self.assertEqual(layer.blobs[0].data[0], -1) net.copy_from(caffemodel_file) self.assertEqual(layer.blobs[0].data[0], 1) + if os.name == 'nt': + # On Windows, attempting to remove a file that is in use + # causes an exception to be raised." + os.close(h) os.remove(caffemodel_file) # Test weight sharing diff --git a/python/caffe/test/test_solver.py b/python/caffe/test/test_solver.py index f618fded8cd..4c1f09666b0 100644 --- a/python/caffe/test/test_solver.py +++ b/python/caffe/test/test_solver.py @@ -13,7 +13,10 @@ def setUp(self): self.num_output = 13 net_f = simple_net_file(self.num_output) f = tempfile.NamedTemporaryFile(mode='w+', delete=False) - f.write("""net: '""" + net_f + """' + net_f_mod = net_f + if os.name == 'nt': + net_f_mod = net_f_mod.replace("\\", "/") + f.write("""net: '""" + net_f_mod + """' test_iter: 10 test_interval: 10 base_lr: 0.01 momentum: 0.9 weight_decay: 0.0005 lr_policy: 'inv' gamma: 0.0001 power: 0.75 display: 100 max_iter: 100 snapshot_after_train: false diff --git a/src/caffe/common.cpp b/src/caffe/common.cpp index 299d67d4bec..cab406fb9b6 100644 --- a/src/caffe/common.cpp +++ b/src/caffe/common.cpp @@ -1,3 +1,8 @@ +#if defined(_MSC_VER) +#include +#define getpid() _getpid() +#endif + #include #include #include @@ -46,7 +51,11 @@ void GlobalInit(int* pargc, char*** pargv) { // Google logging. ::google::InitGoogleLogging(*(pargv)[0]); // Provide a backtrace on segfault. + + // Windows port of glogs doesn't have this function built +#if !defined(_MSC_VER) ::google::InstallFailureSignalHandler(); +#endif } #ifdef CPU_ONLY // CPU-only Caffe. diff --git a/src/caffe/layers/bnll_layer.cu b/src/caffe/layers/bnll_layer.cu index 8df8ef09afe..768a92bba26 100644 --- a/src/caffe/layers/bnll_layer.cu +++ b/src/caffe/layers/bnll_layer.cu @@ -5,7 +5,7 @@ namespace caffe { -const float kBNLL_THRESHOLD = 50.; +__constant__ float kBNLL_THRESHOLD = 50.; template __global__ void BNLLForward(const int n, const Dtype* in, Dtype* out) { diff --git a/src/caffe/test/test_blob.cpp b/src/caffe/test/test_blob.cpp index a9d7d519e45..4e231cdee8b 100644 --- a/src/caffe/test/test_blob.cpp +++ b/src/caffe/test/test_blob.cpp @@ -35,12 +35,14 @@ TYPED_TEST(BlobSimpleTest, TestInitialization) { EXPECT_EQ(this->blob_->count(), 0); } +#if !defined(CPU_ONLY) TYPED_TEST(BlobSimpleTest, TestPointersCPUGPU) { EXPECT_TRUE(this->blob_preshaped_->gpu_data()); EXPECT_TRUE(this->blob_preshaped_->cpu_data()); EXPECT_TRUE(this->blob_preshaped_->mutable_gpu_data()); EXPECT_TRUE(this->blob_preshaped_->mutable_cpu_data()); } +#endif TYPED_TEST(BlobSimpleTest, TestReshape) { this->blob_->Reshape(2, 3, 4, 5); diff --git a/src/caffe/test/test_gradient_based_solver.cpp b/src/caffe/test/test_gradient_based_solver.cpp index 84c6747f61a..e1dac7420cb 100644 --- a/src/caffe/test/test_gradient_based_solver.cpp +++ b/src/caffe/test/test_gradient_based_solver.cpp @@ -177,6 +177,9 @@ class GradientBasedSolverTest : public MultiDeviceTest { proto << "momentum: " << momentum << " "; } MakeTempDir(&snapshot_prefix_); +#if defined(_MSC_VER) + std::replace(snapshot_prefix_.begin(), snapshot_prefix_.end(), '\\', '/'); +#endif proto << "snapshot_prefix: '" << snapshot_prefix_ << "/' "; if (snapshot) { proto << "snapshot: " << num_iters << " "; @@ -508,9 +511,8 @@ class GradientBasedSolverTest : public MultiDeviceTest { for (int i = 0; i < orig_params.size(); ++i) { param_copies[i].reset(new Blob()); const bool kReshape = true; - for (int copy_diff = false; copy_diff <= true; ++copy_diff) { - param_copies[i]->CopyFrom(*orig_params[i], copy_diff, kReshape); - } + param_copies[i]->CopyFrom(*orig_params[i], false/*copy data*/, kReshape); + param_copies[i]->CopyFrom(*orig_params[i], true/*copy diff*/, kReshape); } // Save the solver history @@ -520,9 +522,10 @@ class GradientBasedSolverTest : public MultiDeviceTest { for (int i = 0; i < orig_history.size(); ++i) { history_copies[i].reset(new Blob()); const bool kReshape = true; - for (int copy_diff = false; copy_diff <= true; ++copy_diff) { - history_copies[i]->CopyFrom(*orig_history[i], copy_diff, kReshape); - } + history_copies[i]->CopyFrom(*orig_history[i], + false/*copy data*/, kReshape); + history_copies[i]->CopyFrom(*orig_history[i], + true/*copy diff*/, kReshape); } // Run the solver for num_iters iterations and snapshot. diff --git a/src/caffe/test/test_lrn_layer.cpp b/src/caffe/test/test_lrn_layer.cpp index 4c97b1ae07b..23b52469a8f 100644 --- a/src/caffe/test/test_lrn_layer.cpp +++ b/src/caffe/test/test_lrn_layer.cpp @@ -279,11 +279,10 @@ class CuDNNLRNLayerTest : public GPUDeviceTest { vector*> blob_top_vec_; }; -template -void CuDNNLRNLayerTest::ReferenceLRNForward( - const Blob& blob_bottom, const LayerParameter& layer_param, - Blob* blob_top) { - typedef TypeParam Dtype; +template +void CuDNNLRNLayerTest::ReferenceLRNForward( + const Blob& blob_bottom, const LayerParameter& layer_param, + Blob* blob_top) { blob_top->Reshape(blob_bottom.num(), blob_bottom.channels(), blob_bottom.height(), blob_bottom.width()); Dtype* top_data = blob_top->mutable_cpu_data(); diff --git a/src/caffe/util/db_lmdb.cpp b/src/caffe/util/db_lmdb.cpp index 0bc82b53e2b..cd17447989d 100644 --- a/src/caffe/util/db_lmdb.cpp +++ b/src/caffe/util/db_lmdb.cpp @@ -1,13 +1,27 @@ #ifdef USE_LMDB #include "caffe/util/db_lmdb.hpp" +#if defined(_MSC_VER) +#include +#define mkdir(X, Y) _mkdir(X) +#endif + #include #include namespace caffe { namespace db { +#ifdef _MSC_VER +// On Windows lmdb creates file with the full size causing test failures due +// to insufficient disk space. We will reduce lmdb size to make tests pass. +const size_t LMDB_MAP_SIZE = 104857600; // 100 MB +// Constant will overflow on 32-bit build, assert that we are using correct +// build. +static_assert(sizeof(size_t) >= 8, "LMDB size overflow."); +#else const size_t LMDB_MAP_SIZE = 1099511627776; // 1 TB +#endif void LMDB::Open(const string& source, Mode mode) { MDB_CHECK(mdb_env_create(&mdb_env_)); diff --git a/src/caffe/util/hdf5.cpp b/src/caffe/util/hdf5.cpp index 7730e76ab87..051c9b20bd1 100644 --- a/src/caffe/util/hdf5.cpp +++ b/src/caffe/util/hdf5.cpp @@ -29,31 +29,58 @@ void hdf5_load_nd_dataset_helper( CHECK_GE(status, 0) << "Failed to get dataset info for " << dataset_name_; switch (class_) { case H5T_FLOAT: - LOG_FIRST_N(INFO, 1) << "Datatype class: H5T_FLOAT"; - break; + // In VC++ declaring and initializing variables in case statement without + // curly braces (new scope), cause compiler error C2360 + // https://msdn.microsoft.com/en-us/library/61af7cx3.aspx + { + LOG_FIRST_N(INFO, 1) << "Datatype class: H5T_FLOAT"; + break; + } case H5T_INTEGER: - LOG_FIRST_N(INFO, 1) << "Datatype class: H5T_INTEGER"; - break; + { + LOG_FIRST_N(INFO, 1) << "Datatype class: H5T_INTEGER"; + break; + } case H5T_TIME: - LOG(FATAL) << "Unsupported datatype class: H5T_TIME"; + { + LOG(FATAL) << "Unsupported datatype class: H5T_TIME"; + } case H5T_STRING: - LOG(FATAL) << "Unsupported datatype class: H5T_STRING"; + { + LOG(FATAL) << "Unsupported datatype class: H5T_STRING"; + } case H5T_BITFIELD: - LOG(FATAL) << "Unsupported datatype class: H5T_BITFIELD"; + { + LOG(FATAL) << "Unsupported datatype class: H5T_BITFIELD"; + } case H5T_OPAQUE: - LOG(FATAL) << "Unsupported datatype class: H5T_OPAQUE"; + { + LOG(FATAL) << "Unsupported datatype class: H5T_OPAQUE"; + } case H5T_COMPOUND: - LOG(FATAL) << "Unsupported datatype class: H5T_COMPOUND"; + { + LOG(FATAL) << "Unsupported datatype class: H5T_COMPOUND"; + } case H5T_REFERENCE: - LOG(FATAL) << "Unsupported datatype class: H5T_REFERENCE"; + { + LOG(FATAL) << "Unsupported datatype class: H5T_REFERENCE"; + } case H5T_ENUM: - LOG(FATAL) << "Unsupported datatype class: H5T_ENUM"; + { + LOG(FATAL) << "Unsupported datatype class: H5T_ENUM"; + } case H5T_VLEN: - LOG(FATAL) << "Unsupported datatype class: H5T_VLEN"; + { + LOG(FATAL) << "Unsupported datatype class: H5T_VLEN"; + } case H5T_ARRAY: - LOG(FATAL) << "Unsupported datatype class: H5T_ARRAY"; + { + LOG(FATAL) << "Unsupported datatype class: H5T_ARRAY"; + } default: - LOG(FATAL) << "Datatype class unknown"; + { + LOG(FATAL) << "Datatype class unknown"; + } } vector blob_dims(dims.size()); diff --git a/src/caffe/util/io.cpp b/src/caffe/util/io.cpp index 835d2d4e4ff..f679df8157a 100644 --- a/src/caffe/util/io.cpp +++ b/src/caffe/util/io.cpp @@ -1,4 +1,9 @@ #include + +#if defined(_MSC_VER) +#include +#endif + #include #include #include @@ -50,7 +55,11 @@ void WriteProtoToTextFile(const Message& proto, const char* filename) { } bool ReadProtoFromBinaryFile(const char* filename, Message* proto) { +#if defined (_MSC_VER) // for MSC compiler binary flag needs to be specified + int fd = open(filename, O_RDONLY | O_BINARY); +#else int fd = open(filename, O_RDONLY); +#endif CHECK_NE(fd, -1) << "File not found: " << filename; ZeroCopyInputStream* raw_input = new FileInputStream(fd); CodedInputStream* coded_input = new CodedInputStream(raw_input); diff --git a/tools/caffe.cpp b/tools/caffe.cpp index ebe95d61ef1..7f8dc7d2513 100644 --- a/tools/caffe.cpp +++ b/tools/caffe.cpp @@ -148,6 +148,7 @@ caffe::SolverAction::Enum GetRequestedAction( return caffe::SolverAction::NONE; } LOG(FATAL) << "Invalid signal effect \""<< flag_value << "\" was specified"; + return caffe::SolverAction::NONE; } // Train / Finetune a model. @@ -196,14 +197,19 @@ int train() { Caffe::set_solver_count(gpus.size()); } +#if !defined(_MSC_VER) + // Signals are not properly supported in Windows. caffe::SignalHandler signal_handler( GetRequestedAction(FLAGS_sigint_effect), GetRequestedAction(FLAGS_sighup_effect)); +#endif shared_ptr > solver(caffe::SolverRegistry::CreateSolver(solver_param)); +#if !defined(_MSC_VER) solver->SetActionFunction(signal_handler.GetActionFunction()); +#endif if (FLAGS_snapshot.size()) { LOG(INFO) << "Resuming from " << FLAGS_snapshot; diff --git a/windows/Caffe.sln b/windows/Caffe.sln new file mode 100644 index 00000000000..9807327d263 --- /dev/null +++ b/windows/Caffe.sln @@ -0,0 +1,87 @@ + +Microsoft Visual Studio Solution File, Format Version 12.00 +# Visual Studio 2013 +VisualStudioVersion = 12.0.40629.0 +MinimumVisualStudioVersion = 10.0.40219.1 +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "libcaffe", "libcaffe\libcaffe.vcxproj", "{A9ACEF83-7B63-4574-A554-89CE869EA141}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "caffe", "caffe\caffe.vcxproj", "{CE6BBC46-9EFC-4029-9065-85A023866AFB}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "compute_image_mean", "compute_image_mean\compute_image_mean.vcxproj", "{09A8EDAC-20B9-414F-9654-961388FD5A8C}" + ProjectSection(ProjectDependencies) = postProject + {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} + EndProjectSection +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "convert_imageset", "convert_imageset\convert_imageset.vcxproj", "{44AAEF8E-2DF2-4534-AD6C-50017997897B}" + ProjectSection(ProjectDependencies) = postProject + {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} + EndProjectSection +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "extract_features", "extract_features\extract_features.vcxproj", "{C4A4173A-1BBA-4668-B506-0538A7D259E4}" + ProjectSection(ProjectDependencies) = postProject + {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} + EndProjectSection +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "test_all", "test_all\test_all.vcxproj", "{00BBA8C0-707D-42A7-82FF-D5211185ED7F}" + ProjectSection(ProjectDependencies) = postProject + {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} + EndProjectSection +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pycaffe", "pycaffe\pycaffe.vcxproj", "{38B6CE09-4B1A-4E72-A547-8A3299D8DA60}" + ProjectSection(ProjectDependencies) = postProject + {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} + EndProjectSection +EndProject +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "props", "props", "{632DD6E1-28DF-42F9-AD7F-1C1F2D38765C}" + ProjectSection(SolutionItems) = preProject + CommonSettings.props = CommonSettings.props + EndProjectSection +EndProject +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "scripts", "scripts", "{E2EF4AB6-AB52-4777-9783-4669A0D61F80}" + ProjectSection(SolutionItems) = preProject + scripts\BinplaceCudaDependencies.cmd = scripts\BinplaceCudaDependencies.cmd + scripts\FixGFlagsNaming.cmd = scripts\FixGFlagsNaming.cmd + scripts\ProtoCompile.cmd = scripts\ProtoCompile.cmd + scripts\PythonPostBuild.cmd = scripts\PythonPostBuild.cmd + scripts\PythonPreBuild.cmd = scripts\PythonPreBuild.cmd + EndProjectSection +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|x64 = Debug|x64 + Release|x64 = Release|x64 + EndGlobalSection + GlobalSection(ProjectConfigurationPlatforms) = postSolution + {A9ACEF83-7B63-4574-A554-89CE869EA141}.Debug|x64.ActiveCfg = Debug|x64 + {A9ACEF83-7B63-4574-A554-89CE869EA141}.Debug|x64.Build.0 = Debug|x64 + {A9ACEF83-7B63-4574-A554-89CE869EA141}.Release|x64.ActiveCfg = Release|x64 + {A9ACEF83-7B63-4574-A554-89CE869EA141}.Release|x64.Build.0 = Release|x64 + {CE6BBC46-9EFC-4029-9065-85A023866AFB}.Debug|x64.ActiveCfg = Debug|x64 + {CE6BBC46-9EFC-4029-9065-85A023866AFB}.Debug|x64.Build.0 = Debug|x64 + {CE6BBC46-9EFC-4029-9065-85A023866AFB}.Release|x64.ActiveCfg = Release|x64 + {CE6BBC46-9EFC-4029-9065-85A023866AFB}.Release|x64.Build.0 = Release|x64 + {09A8EDAC-20B9-414F-9654-961388FD5A8C}.Debug|x64.ActiveCfg = Debug|x64 + {09A8EDAC-20B9-414F-9654-961388FD5A8C}.Debug|x64.Build.0 = Debug|x64 + {09A8EDAC-20B9-414F-9654-961388FD5A8C}.Release|x64.ActiveCfg = Release|x64 + {09A8EDAC-20B9-414F-9654-961388FD5A8C}.Release|x64.Build.0 = Release|x64 + {44AAEF8E-2DF2-4534-AD6C-50017997897B}.Debug|x64.ActiveCfg = Debug|x64 + {44AAEF8E-2DF2-4534-AD6C-50017997897B}.Debug|x64.Build.0 = Debug|x64 + {44AAEF8E-2DF2-4534-AD6C-50017997897B}.Release|x64.ActiveCfg = Release|x64 + {44AAEF8E-2DF2-4534-AD6C-50017997897B}.Release|x64.Build.0 = Release|x64 + {C4A4173A-1BBA-4668-B506-0538A7D259E4}.Debug|x64.ActiveCfg = Debug|x64 + {C4A4173A-1BBA-4668-B506-0538A7D259E4}.Debug|x64.Build.0 = Debug|x64 + {C4A4173A-1BBA-4668-B506-0538A7D259E4}.Release|x64.ActiveCfg = Release|x64 + {C4A4173A-1BBA-4668-B506-0538A7D259E4}.Release|x64.Build.0 = Release|x64 + {00BBA8C0-707D-42A7-82FF-D5211185ED7F}.Debug|x64.ActiveCfg = Debug|x64 + {00BBA8C0-707D-42A7-82FF-D5211185ED7F}.Debug|x64.Build.0 = Debug|x64 + {00BBA8C0-707D-42A7-82FF-D5211185ED7F}.Release|x64.ActiveCfg = Release|x64 + {00BBA8C0-707D-42A7-82FF-D5211185ED7F}.Release|x64.Build.0 = Release|x64 + {38B6CE09-4B1A-4E72-A547-8A3299D8DA60}.Debug|x64.ActiveCfg = Debug|x64 + {38B6CE09-4B1A-4E72-A547-8A3299D8DA60}.Debug|x64.Build.0 = Debug|x64 + {38B6CE09-4B1A-4E72-A547-8A3299D8DA60}.Release|x64.ActiveCfg = Release|x64 + {38B6CE09-4B1A-4E72-A547-8A3299D8DA60}.Release|x64.Build.0 = Release|x64 + EndGlobalSection + GlobalSection(SolutionProperties) = preSolution + HideSolutionNode = FALSE + EndGlobalSection +EndGlobal diff --git a/windows/CommonSettings.props.example b/windows/CommonSettings.props.example new file mode 100644 index 00000000000..f5b4f3a7377 --- /dev/null +++ b/windows/CommonSettings.props.example @@ -0,0 +1,99 @@ + + + + + $(SolutionDir)..\Build + + false + true + 7.5 + + false + + + + compute_35,sm_35 + + + + $(SolutionDir)..\..\CaffeCuDnn + $(SolutionDir)\scripts + + + cublas.lib;cuda.lib;curand.lib;cudart.lib + + + cudnn.lib;$(CudaDependencies) + $(CuDnnPath)\cuda\lib\x64;$(LibraryPath) + $(CuDnnPath)\cuda\include;$(IncludePath) + + + $(BuildDir)\$(Platform)\$(Configuration)\ + $(BuildDir)\Int\$(ProjectName)\$(Platform)\$(Configuration)\ + + + $(OutDir);$(CUDA_PATH)\lib\$(Platform);$(LibraryPath) + $(SolutionDir)..\include;$(SolutionDir)..\include\caffe\proto;$(CUDA_PATH)\include;$(IncludePath) + + + C:\Miniconda2\ + $(PythonDir)\libs;$(LibraryPath) + $(PythonDir)\include;$(IncludePath) + + + + CPU_ONLY;%(PreprocessorDefinitions) + + + + + USE_CUDNN;%(PreprocessorDefinitions) + + + USE_CUDNN + + + + + WITH_PYTHON_LAYER;BOOST_PYTHON_STATIC_LIB;%(PreprocessorDefinitions) + + + + + false + true + _SCL_SECURE_NO_WARNINGS;USE_OPENCV;USE_LEVELDB;USE_LMDB;%(PreprocessorDefinitions) + true + + + + + Full + NDEBUG;%(PreprocessorDefinitions) + MultiThreadedDLL + true + + + true + true + UseLinkTimeCodeGeneration + true + + + + + Disabled + _DEBUG;%(PreprocessorDefinitions) + MultiThreadedDebugDLL + + + true + + + \ No newline at end of file diff --git a/windows/CommonSettings.targets b/windows/CommonSettings.targets new file mode 100644 index 00000000000..b9077d354b7 --- /dev/null +++ b/windows/CommonSettings.targets @@ -0,0 +1,11 @@ + + + + + + + + + \ No newline at end of file diff --git a/windows/caffe/caffe.vcxproj b/windows/caffe/caffe.vcxproj new file mode 100644 index 00000000000..564dbbee645 --- /dev/null +++ b/windows/caffe/caffe.vcxproj @@ -0,0 +1,118 @@ + + + + + + + + Debug + x64 + + + Release + x64 + + + + {CE6BBC46-9EFC-4029-9065-85A023866AFB} + Win32Proj + x64 + caffe + e703126e + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + $(ScriptsDir)\FixGFlagsNaming.cmd "$(OutDir)" $(Configuration) + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + $(ScriptsDir)\FixGFlagsNaming.cmd "$(OutDir)" $(Configuration) + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/caffe/packages.config b/windows/caffe/packages.config new file mode 100644 index 00000000000..ff68ac185a6 --- /dev/null +++ b/windows/caffe/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/compute_image_mean/compute_image_mean.vcxproj b/windows/compute_image_mean/compute_image_mean.vcxproj new file mode 100644 index 00000000000..a41ec77b614 --- /dev/null +++ b/windows/compute_image_mean/compute_image_mean.vcxproj @@ -0,0 +1,112 @@ + + + + + + + + Debug + x64 + + + Release + x64 + + + + {09A8EDAC-20B9-414F-9654-961388FD5A8C} + Win32Proj + x64 + compute_image_mean + f6e60ad8 + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/compute_image_mean/packages.config b/windows/compute_image_mean/packages.config new file mode 100644 index 00000000000..ff68ac185a6 --- /dev/null +++ b/windows/compute_image_mean/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/convert_imageset/convert_imageset.vcxproj b/windows/convert_imageset/convert_imageset.vcxproj new file mode 100644 index 00000000000..7b91235de54 --- /dev/null +++ b/windows/convert_imageset/convert_imageset.vcxproj @@ -0,0 +1,112 @@ + + + + + + + + Debug + x64 + + + Release + x64 + + + + {44AAEF8E-2DF2-4534-AD6C-50017997897B} + Win32Proj + x64 + convert_imageset + aa5aeccc + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/convert_imageset/packages.config b/windows/convert_imageset/packages.config new file mode 100644 index 00000000000..ff68ac185a6 --- /dev/null +++ b/windows/convert_imageset/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/extract_features/extract_features.vcxproj b/windows/extract_features/extract_features.vcxproj new file mode 100644 index 00000000000..c251edf8994 --- /dev/null +++ b/windows/extract_features/extract_features.vcxproj @@ -0,0 +1,118 @@ + + + + + + + + Debug + x64 + + + Release + x64 + + + + {C4A4173A-1BBA-4668-B506-0538A7D259E4} + Win32Proj + x64 + extract_features + 59a71837 + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + 4005 + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + 4005 + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/extract_features/packages.config b/windows/extract_features/packages.config new file mode 100644 index 00000000000..ff68ac185a6 --- /dev/null +++ b/windows/extract_features/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/libcaffe/libcaffe.vcxproj b/windows/libcaffe/libcaffe.vcxproj new file mode 100644 index 00000000000..6488ff0a3a6 --- /dev/null +++ b/windows/libcaffe/libcaffe.vcxproj @@ -0,0 +1,383 @@ + + + + + + + + + Debug + x64 + + + Release + x64 + + + + {A9ACEF83-7B63-4574-A554-89CE869EA141} + libcaffe + v120 + + + + StaticLibrary + true + Unicode + + + StaticLibrary + false + true + Unicode + + + + + + + b4efcc07 + + + + + + + + true + Console + + + "$(ScriptsDir)\ProtoCompile.cmd" "$(SolutionDir)" "$(ProtocDir)" + + + "$(ScriptsDir)\BinplaceCudaDependencies.cmd" "$(CudaToolkitBinDir)" "$(CuDnnPath)" $(CpuOnlyBuild) $(UseCuDNN) "$(OutDir)" + + + 64 + $(CudaArchitecture) + true + -Xcudafe "--diag_suppress=exception_spec_override_incompat --diag_suppress=useless_using_declaration --diag_suppress=field_without_dll_interface" -D_SCL_SECURE_NO_WARNINGS -DGFLAGS_DLL_DECL= + + + 4661;4005;4812;4715;%(DisableSpecificWarnings) + $(ProjectDir)\..\..\src\;%(AdditionalIncludeDirectories) + + + /ignore:4221 %(AdditionalOptions) + + + + + Console + + + "$(ScriptsDir)\ProtoCompile.cmd" "$(SolutionDir)" "$(ProtocDir)" + + + "$(ScriptsDir)\BinplaceCudaDependencies.cmd" "$(CudaToolkitBinDir)" "$(CuDnnPath)" $(CpuOnlyBuild) $(UseCuDNN) "$(OutDir)" + + + 64 + $(CudaArchitecture) + -Xcudafe "--diag_suppress=exception_spec_override_incompat --diag_suppress=useless_using_declaration --diag_suppress=field_without_dll_interface" -D_SCL_SECURE_NO_WARNINGS -DGFLAGS_DLL_DECL= + + + 4661;4005;4812;4715;%(DisableSpecificWarnings) + $(ProjectDir)\..\..\src\;%(AdditionalIncludeDirectories) + + + /ignore:4221 %(AdditionalOptions) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/libcaffe/libcaffe.vcxproj.filters b/windows/libcaffe/libcaffe.vcxproj.filters new file mode 100644 index 00000000000..ef71751c985 --- /dev/null +++ b/windows/libcaffe/libcaffe.vcxproj.filters @@ -0,0 +1,794 @@ + + + + + {253af030-e1e0-426c-9a22-6315b0d2dab7} + + + {36c36b62-e801-40f2-bba9-a79f09fa4dba} + + + {66b19093-f1ad-443e-b5d3-f55955ff0ae2} + + + {3be25bf1-cf46-47da-b1ff-30cb442da7c5} + + + {9e47fb53-4e3b-4e03-b677-a58cc26af7fb} + + + {bbb6f6f1-8a55-469b-8729-a61f87d6b63d} + + + {f9e33710-c82c-4808-90e7-96620a190b3c} + + + {9a64cba7-8bef-4df3-b933-adec019daadb} + + + {96fba2c6-dad0-4766-b354-08a7768d57d8} + + + {e4995612-1b91-40ea-9756-44382eddca40} + + + {c820c58e-d861-4d88-8b18-2180996d0657} + + + {f10cfd17-81b6-4a08-829d-1a1fa4769d2e} + + + {fcb8114c-3425-41da-b30a-af2cb33dd851} + + + + + src\util + + + src\util + + + src\util + + + src\util + + + src\util + + + src\util + + + src\proto + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src\util + + + src\util + + + src\util + + + src\util + + + src\util + + + src + + + src + + + src + + + src\util + + + src\solvers + + + src\solvers + + + src\solvers + + + src\solvers + + + src\solvers + + + src\solvers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + src\layers + + + + + include\proto + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\util + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include\layers + + + include + + + include + + + include + + + include + + + include + + + include + + + include + + + include + + + include + + + include + + + include + + + include + + + include + + + include + + + include + + + include\layers + + + include\layers + + + include\layers + + + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\util + + + cu\util + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\layers + + + cu\solvers + + + cu\solvers + + + cu\solvers + + + cu\solvers + + + cu\solvers + + + cu\solvers + + + + + + \ No newline at end of file diff --git a/windows/libcaffe/packages.config b/windows/libcaffe/packages.config new file mode 100644 index 00000000000..ab2d5ffa952 --- /dev/null +++ b/windows/libcaffe/packages.config @@ -0,0 +1,14 @@ + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/nuget.config b/windows/nuget.config new file mode 100644 index 00000000000..fc77aae0d3f --- /dev/null +++ b/windows/nuget.config @@ -0,0 +1,4 @@ + + + ..\..\NugetPackages + \ No newline at end of file diff --git a/windows/pycaffe/packages.config b/windows/pycaffe/packages.config new file mode 100644 index 00000000000..15803452a2e --- /dev/null +++ b/windows/pycaffe/packages.config @@ -0,0 +1,19 @@ + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/pycaffe/pycaffe.vcxproj b/windows/pycaffe/pycaffe.vcxproj new file mode 100644 index 00000000000..e48a61cdb34 --- /dev/null +++ b/windows/pycaffe/pycaffe.vcxproj @@ -0,0 +1,129 @@ + + + + + + + + + Debug + x64 + + + Release + x64 + + + + {38B6CE09-4B1A-4E72-A547-8A3299D8DA60} + pycaffe + + + + v120 + DynamicLibrary + + + + + + + + + .pyd + _caffe + + + $(PythonDir)\Lib\site-packages\numpy\core\include\;$(IncludePath) + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + + + + + 4003 + + + "$(ScriptsDir)\PythonPreBuild.cmd" "$(SolutionDir)" "$(ProtocDir)" "$(OutDir)" + + + "$(ScriptsDir)\PythonPostBuild.cmd" "$(SolutionDir)" "$(OutDir)" + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + $(BuildDependsOn) + OriginalBuild;SkipBuild + 14b5f2c8 + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/scripts/BinplaceCudaDependencies.cmd b/windows/scripts/BinplaceCudaDependencies.cmd new file mode 100644 index 00000000000..4f5b0e480f7 --- /dev/null +++ b/windows/scripts/BinplaceCudaDependencies.cmd @@ -0,0 +1,22 @@ +set CUDA_TOOLKIT_BIN_DIR=%~1% +set CUDNN_PATH=%~2% +set IS_CPU_ONLY_BUILD=%3% +set USE_CUDNN=%4% +set OUTPUT_DIR=%~5% + +if %IS_CPU_ONLY_BUILD% == true ( + echo BinplaceCudaDependencies : CPU only build, don't copy cuda dependencies. + ) else ( + echo BinplaceCudaDependencies : Copy cudart*.dll, cublas*dll, curand*.dll to output. + + copy /y "%CUDA_TOOLKIT_BIN_DIR%\cudart*.dll" "%OUTPUT_DIR%" + copy /y "%CUDA_TOOLKIT_BIN_DIR%\cublas*.dll" "%OUTPUT_DIR%" + copy /y "%CUDA_TOOLKIT_BIN_DIR%\curand*.dll" "%OUTPUT_DIR%" + + if %USE_CUDNN% == true ( + echo BinplaceCudaDependencies : Copy cunn*.dll to output. + copy /y "%CUDNN_PATH%\cuda\bin\cudnn*.dll" "%OUTPUT_DIR%" + ) else ( + echo BinplaceCudaDependencies : cuDNN isn't enabled. + ) +) \ No newline at end of file diff --git a/windows/scripts/FixGFlagsNaming.cmd b/windows/scripts/FixGFlagsNaming.cmd new file mode 100644 index 00000000000..2dc113325ab --- /dev/null +++ b/windows/scripts/FixGFlagsNaming.cmd @@ -0,0 +1,24 @@ +:: Glog nuget package has dependency on GFlags nuget package +:: Caffe also has direct dependency on GFlags +:: Unfortunately in GLog nuget package, dependency to GFlags dll was incorrectly set (naming is wrong) +:: For this reasons Caffe needs gflags.dll/gflagsd.dll in release/debug +:: and GLog needs libgflags.dll/libgflags-debug.dll in release/debug +:: This scripts is a workaround for this issue. + +set OUTPUT_DIR=%~1% +set BUILD_CONFIG=%2% + +if %BUILD_CONFIG% == Release ( + set originalDllName=gflags.dll + set newDllName=libgflags.dll +) else ( + set originalDllName=gflagsd.dll + set newDllName=libgflags-debug.dll +) + +if exist "%OUTPUT_DIR%\%newDllName%" ( + echo FixGFlagsNaming.cmd : "%newDllName%" already exists +) else ( + echo FixGFlagsNaming.cmd : mklink /H "%OUTPUT_DIR%\%newDllName%" "%OUTPUT_DIR%\%originalDllName%" + mklink /H "%OUTPUT_DIR%\%newDllName%" "%OUTPUT_DIR%\%originalDllName%" +) \ No newline at end of file diff --git a/windows/scripts/ProtoCompile.cmd b/windows/scripts/ProtoCompile.cmd new file mode 100644 index 00000000000..d056e6a17c0 --- /dev/null +++ b/windows/scripts/ProtoCompile.cmd @@ -0,0 +1,27 @@ +set SOLUTION_DIR=%~1% +set PROTO_DIR=%~2% + +set INCLUDE_PROTO_DIR=%SOLUTION_DIR%..\include\caffe\proto +SET SRC_PROTO_DIR=%SOLUTION_DIR%..\src\caffe\proto +set PROTO_TEMP_DIR=%SRC_PROTO_DIR%\temp + +echo ProtoCompile.cmd : Create proto temp directory "%PROTO_TEMP_DIR%" +mkdir "%PROTO_TEMP_DIR%" + +echo ProtoCompile.cmd : Generating "%PROTO_TEMP_DIR%\caffe.pb.h" and "%PROTO_TEMP_DIR%\caffe.pb.cc" +"%PROTO_DIR%protoc" --proto_path="%SRC_PROTO_DIR%" --cpp_out="%PROTO_TEMP_DIR%" "%SRC_PROTO_DIR%\caffe.proto" + +echo ProtoCompile.cmd : Create proto include directory +mkdir "%INCLUDE_PROTO_DIR%" + +echo ProtoCompile.cmd : Compare newly compiled caffe.pb.h with existing one +fc /b "%PROTO_TEMP_DIR%\caffe.pb.h" "%INCLUDE_PROTO_DIR%\caffe.pb.h" > NUL + +if errorlevel 1 ( + echo ProtoCompile.cmd : Move newly generated caffe.pb.h to "%INCLUDE_PROTO_DIR%\caffe.pb.h" + echo ProtoCompile.cmd : and caffe.pb.cc to "%SRC_PROTO_DIR%\caffe.pb.cc" + move /y "%PROTO_TEMP_DIR%\caffe.pb.h" "%INCLUDE_PROTO_DIR%\caffe.pb.h" + move /y "%PROTO_TEMP_DIR%\caffe.pb.cc" "%SRC_PROTO_DIR%\caffe.pb.cc" +) + +rmdir /S /Q "%PROTO_TEMP_DIR%" \ No newline at end of file diff --git a/windows/scripts/PythonPostBuild.cmd b/windows/scripts/PythonPostBuild.cmd new file mode 100644 index 00000000000..6eb3aa759d8 --- /dev/null +++ b/windows/scripts/PythonPostBuild.cmd @@ -0,0 +1,8 @@ +set SOLUTION_DIR=%~1% +set OUTPUT_DIR=%~2% + +echo PythonPostBuild.cmd : copy python generated scripts to output. + +copy /y "%SOLUTION_DIR%..\python\caffe\*.py" "%OUTPUT_DIR%pycaffe\caffe" +copy /y "%SOLUTION_DIR%..\python\*.py" "%OUTPUT_DIR%pycaffe" +move /y "%OUTPUT_DIR%_caffe.*" "%OUTPUT_DIR%pycaffe\caffe" \ No newline at end of file diff --git a/windows/scripts/PythonPreBuild.cmd b/windows/scripts/PythonPreBuild.cmd new file mode 100644 index 00000000000..1f07b1d2f3b --- /dev/null +++ b/windows/scripts/PythonPreBuild.cmd @@ -0,0 +1,15 @@ +set SOLUTION_DIR=%~1% +set PROTO_COMPILER_DIR=%~2% +set OUTPUT_DIR=%~3% + +echo PythonPreBuild.cmd : Create output directories for python scripts. + +if not exist "%OUTPUT_DIR%\pycaffe" mkdir "%OUTPUT_DIR%\pycaffe" +if not exist "%OUTPUT_DIR%\pycaffe\caffe" mkdir "%OUTPUT_DIR%\pycaffe\caffe" +if not exist "%OUTPUT_DIR%\pycaffe\caffe\proto" mkdir "%OUTPUT_DIR%\pycaffe\caffe\proto" + +echo PythonPreBuild.cmd : Create dummy __init__.py file +rem. > "%OUTPUT_DIR%\pycaffe\caffe\proto\__init__.py" + +echo PythonPreBuild.cmd : Generating src\caffe\proto\caffe.pb.h with python bindings +"%PROTO_COMPILER_DIR%\protoc" "%SOLUTION_DIR%\..\src\caffe\proto\caffe.proto" --proto_path="%SOLUTION_DIR%\..\src\caffe\proto" --python_out="%OUTPUT_DIR%\pycaffe\caffe\proto" \ No newline at end of file diff --git a/windows/test_all/packages.config b/windows/test_all/packages.config new file mode 100644 index 00000000000..ff68ac185a6 --- /dev/null +++ b/windows/test_all/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/test_all/test_all.vcxproj b/windows/test_all/test_all.vcxproj new file mode 100644 index 00000000000..aa3409a8679 --- /dev/null +++ b/windows/test_all/test_all.vcxproj @@ -0,0 +1,207 @@ + + + + + + + + + Debug + x64 + + + Release + x64 + + + + {00BBA8C0-707D-42A7-82FF-D5211185ED7F} + Win32Proj + x64 + test_all + f6a28848 + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + 4005;%(DisableSpecificWarnings) + $(ProjectDir)\..\..\src;%(AdditionalIncludeDirectories) + + + 64 + $(CudaArchitecture) + true + -Xcudafe "--diag_suppress=exception_spec_override_incompat --diag_suppress=useless_using_declaration --diag_suppress=field_without_dll_interface --diag_suppress=boolean_controlling_expr_is_constant" -D_SCL_SECURE_NO_WARNINGS -DGFLAGS_DLL_DECL= + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + 4005;%(DisableSpecificWarnings) + $(ProjectDir)\..\..\src;%(AdditionalIncludeDirectories) + + + 64 + $(CudaArchitecture) + -Xcudafe "--diag_suppress=exception_spec_override_incompat --diag_suppress=useless_using_declaration --diag_suppress=field_without_dll_interface --diag_suppress=boolean_controlling_expr_is_constant" -D_SCL_SECURE_NO_WARNINGS -DGFLAGS_DLL_DECL= + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Document + + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/test_all/test_all.vcxproj.filters b/windows/test_all/test_all.vcxproj.filters new file mode 100644 index 00000000000..1e2f107c956 --- /dev/null +++ b/windows/test_all/test_all.vcxproj.filters @@ -0,0 +1,232 @@ + + + + + {4FC737F1-C7A5-4376-A066-2A32D752A2FF} + cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx + + + {93995380-89BD-4b04-88EB-625FBE52EBFB} + h;hh;hpp;hxx;hm;inl;inc;xsd + + + {46116906-a399-42c7-be9d-8a20cbbb0169} + + + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + src + + + + + include + + + include + + + + + + + + cu + + + \ No newline at end of file From f2f0a0d02feab2d291fbffdc87f34ae5fd7ade52 Mon Sep 17 00:00:00 2001 From: Evan Shelhamer Date: Wed, 24 Feb 2016 13:32:23 -0800 Subject: [PATCH 02/51] make branch README for Windows port --- README.md | 38 +++++--------------------------------- 1 file changed, 5 insertions(+), 33 deletions(-) diff --git a/README.md b/README.md index 44b9e62c157..2b5e69ab33d 100644 --- a/README.md +++ b/README.md @@ -1,37 +1,9 @@ -# Caffe +# Windows Caffe -[![Build Status](https://travis-ci.org/BVLC/caffe.svg?branch=master)](https://travis-ci.org/BVLC/caffe) -[![License](https://img.shields.io/badge/license-BSD-blue.svg)](LICENSE) +**This is an experimental, Microsoft-led branch by Pavle Josipovic (@pavlejosipovic). It is a work-in-progress.** -Caffe is a deep learning framework made with expression, speed, and modularity in mind. -It is developed by the Berkeley Vision and Learning Center ([BVLC](http://bvlc.eecs.berkeley.edu)) and community contributors. +This branch of Caffe ports the framework to Windows. -Check out the [project site](http://caffe.berkeleyvision.org) for all the details like +## Further Details -- [DIY Deep Learning for Vision with Caffe](https://docs.google.com/presentation/d/1UeKXVgRvvxg9OUdh_UiC5G71UMscNPlvArsWER41PsU/edit#slide=id.p) -- [Tutorial Documentation](http://caffe.berkeleyvision.org/tutorial/) -- [BVLC reference models](http://caffe.berkeleyvision.org/model_zoo.html) and the [community model zoo](https://github.com/BVLC/caffe/wiki/Model-Zoo) -- [Installation instructions](http://caffe.berkeleyvision.org/installation.html) - -and step-by-step examples. - -[![Join the chat at https://gitter.im/BVLC/caffe](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/BVLC/caffe?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) - -Please join the [caffe-users group](https://groups.google.com/forum/#!forum/caffe-users) or [gitter chat](https://gitter.im/BVLC/caffe) to ask questions and talk about methods and models. -Framework development discussions and thorough bug reports are collected on [Issues](https://github.com/BVLC/caffe/issues). - -Happy brewing! - -## License and Citation - -Caffe is released under the [BSD 2-Clause license](https://github.com/BVLC/caffe/blob/master/LICENSE). -The BVLC reference models are released for unrestricted use. - -Please cite Caffe in your publications if it helps your research: - - @article{jia2014caffe, - Author = {Jia, Yangqing and Shelhamer, Evan and Donahue, Jeff and Karayev, Sergey and Long, Jonathan and Girshick, Ross and Guadarrama, Sergio and Darrell, Trevor}, - Journal = {arXiv preprint arXiv:1408.5093}, - Title = {Caffe: Convolutional Architecture for Fast Feature Embedding}, - Year = {2014} - } +Refer to the BVLC/caffe master branch README for all other details such as license, citation, and so on. From bed9df09ac7d0215d9915f60572e98336cee952e Mon Sep 17 00:00:00 2001 From: pavlejosipovic Date: Tue, 1 Mar 2016 11:03:57 +0100 Subject: [PATCH 03/51] Update README.md Add build badges. Add windows build guide. --- README.md | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/README.md b/README.md index 2b5e69ab33d..00b9f5371fc 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,47 @@ This branch of Caffe ports the framework to Windows. +[![Travis Build Status](https://api.travis-ci.org/BVLC/caffe.svg?branch=windows)](https://travis-ci.org/BVLC/caffe) Travis (Linux build) + +[![Build status](https://ci.appveyor.com/api/projects/status/128eg95svel2a2xs?svg=true)] +(https://ci.appveyor.com/project/pavlejosipovic/caffe-v45qi) AppVeyor (Windows build) + +## Windows Setup +**Requirements**: Visual Studio 2013 + +### Pre-Build Steps +Copy `.\windows\CommonSettings.props.example` to `.\windows\CommonSettings.props` + +By defaults Windows build requires `CUDA` and `cuDNN` libraries. +Both can be disabled by adjusting build variables in `.\windows\CommonSettings.props`. +Python support is disabled by default, but can be enabled via `.\windows\CommonSettings.props` as well. +3rd party dependencies required by Caffe are automatically resolved via NuGet. + +### CUDA +Download `CUDA Toolkit 7.5` [from nVidia website](https://developer.nvidia.com/cuda-toolkit). +If you don't have CUDA installed, you can experiment with CPU_ONLY build. +In `.\windows\CommonSettings.props` set `CpuOnlyBuild` to `true` and set `UseCuDNN` to `false`. + +### cuDNN +Download `cuDNN v3` or `cuDNN v4` [from nVidia website](https://developer.nvidia.com/cudnn). +Unpack downloaded zip to `CuDnnPath` defined in `.\windows\CommonSettings.props`. +Alternatively, you can disable cuDNN by setting `UseCuDNN` to `false` in the property file. + +### Python +To build Caffe Python wrapper set `PythonSupport` to `true` in `.\windows\CommonSettings.props`. +Download Miniconda 2.7 64-bit Windows installer [from Miniconda website] (http://conda.pydata.org/miniconda.html). +Install for all users and add Python to PATH (through installer). + +Run the following commands from elevated command prompt: + +``` +conda install --yes numpy scipy matplotlib scikit-image pip +pip install protobuf +``` + +### Build +Now, you should be able to build `.\windows\Caffe.sln` + ## Further Details Refer to the BVLC/caffe master branch README for all other details such as license, citation, and so on. From 14f17f949990762c7d67d83eb7b9ce63df7b465a Mon Sep 17 00:00:00 2001 From: happynear Date: Fri, 26 Feb 2016 10:38:49 +0800 Subject: [PATCH 04/51] support signal handler --- src/caffe/util/signal_handler.cpp | 26 ++++++++++++++++++++++++-- tools/caffe.cpp | 5 ----- windows/libcaffe/libcaffe.vcxproj | 1 + windows/libcaffe/libcaffe.vcxproj.filters | 3 +++ 4 files changed, 28 insertions(+), 7 deletions(-) diff --git a/src/caffe/util/signal_handler.cpp b/src/caffe/util/signal_handler.cpp index 5d764ec524f..6599db47159 100644 --- a/src/caffe/util/signal_handler.cpp +++ b/src/caffe/util/signal_handler.cpp @@ -13,9 +13,15 @@ namespace { void handle_signal(int signal) { switch (signal) { +#ifdef _MSC_VER + case SIGBREAK: // there is no SIGHUP in windows, take SIGBREAK instead. + got_sighup = true; + break; +#else case SIGHUP: got_sighup = true; break; +#endif case SIGINT: got_sigint = true; break; @@ -27,7 +33,14 @@ namespace { LOG(FATAL) << "Tried to hookup signal handlers more than once."; } already_hooked_up = true; - +#ifdef _MSC_VER + if (signal(SIGBREAK, handle_signal) == SIG_ERR) { + LOG(FATAL) << "Cannot install SIGBREAK handler."; + } + if (signal(SIGINT, handle_signal) == SIG_ERR) { + LOG(FATAL) << "Cannot install SIGINT handler."; + } +#else struct sigaction sa; // Setup the handler sa.sa_handler = &handle_signal; @@ -42,11 +55,20 @@ namespace { if (sigaction(SIGINT, &sa, NULL) == -1) { LOG(FATAL) << "Cannot install SIGINT handler."; } +#endif } // Set the signal handlers to the default. void UnhookHandler() { if (already_hooked_up) { +#ifdef _MSC_VER + if (signal(SIGBREAK, SIG_DFL) == SIG_ERR) { + LOG(FATAL) << "Cannot uninstall SIGBREAK handler."; + } + if (signal(SIGINT, SIG_DFL) == SIG_ERR) { + LOG(FATAL) << "Cannot uninstall SIGINT handler."; + } +#else struct sigaction sa; // Setup the sighub handler sa.sa_handler = SIG_DFL; @@ -61,7 +83,7 @@ namespace { if (sigaction(SIGINT, &sa, NULL) == -1) { LOG(FATAL) << "Cannot uninstall SIGINT handler."; } - +#endif already_hooked_up = false; } } diff --git a/tools/caffe.cpp b/tools/caffe.cpp index 7f8dc7d2513..4b760bd0ae9 100644 --- a/tools/caffe.cpp +++ b/tools/caffe.cpp @@ -197,19 +197,14 @@ int train() { Caffe::set_solver_count(gpus.size()); } -#if !defined(_MSC_VER) - // Signals are not properly supported in Windows. caffe::SignalHandler signal_handler( GetRequestedAction(FLAGS_sigint_effect), GetRequestedAction(FLAGS_sighup_effect)); -#endif shared_ptr > solver(caffe::SolverRegistry::CreateSolver(solver_param)); -#if !defined(_MSC_VER) solver->SetActionFunction(signal_handler.GetActionFunction()); -#endif if (FLAGS_snapshot.size()) { LOG(INFO) << "Resuming from " << FLAGS_snapshot; diff --git a/windows/libcaffe/libcaffe.vcxproj b/windows/libcaffe/libcaffe.vcxproj index 6488ff0a3a6..2a0679f2b3f 100644 --- a/windows/libcaffe/libcaffe.vcxproj +++ b/windows/libcaffe/libcaffe.vcxproj @@ -184,6 +184,7 @@ + diff --git a/windows/libcaffe/libcaffe.vcxproj.filters b/windows/libcaffe/libcaffe.vcxproj.filters index ef71751c985..f68cb502aef 100644 --- a/windows/libcaffe/libcaffe.vcxproj.filters +++ b/windows/libcaffe/libcaffe.vcxproj.filters @@ -324,6 +324,9 @@ src\layers + + src\util + From 55f3cfd46d2bf38b7f41343e59e44081ea4481a1 Mon Sep 17 00:00:00 2001 From: Pavle Josipovic Date: Fri, 26 Feb 2016 17:51:17 +0100 Subject: [PATCH 05/51] Tweak default build options Make compiling Maxwell and Kepler cuda arch default. Support extrating cuDNN files to CUDA install folder. Copy caffe dependancies to python output folder. Adjust README.md --- README.md | 13 ++++++++++--- windows/CommonSettings.props.example | 14 +++++++------- windows/scripts/BinplaceCudaDependencies.cmd | 9 +++++++-- windows/scripts/PythonPostBuild.cmd | 3 ++- 4 files changed, 26 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 00b9f5371fc..5582657a7c2 100644 --- a/README.md +++ b/README.md @@ -26,9 +26,11 @@ If you don't have CUDA installed, you can experiment with CPU_ONLY build. In `.\windows\CommonSettings.props` set `CpuOnlyBuild` to `true` and set `UseCuDNN` to `false`. ### cuDNN -Download `cuDNN v3` or `cuDNN v4` [from nVidia website](https://developer.nvidia.com/cudnn). -Unpack downloaded zip to `CuDnnPath` defined in `.\windows\CommonSettings.props`. -Alternatively, you can disable cuDNN by setting `UseCuDNN` to `false` in the property file. +Download `cuDNN v3` or `cuDNN v4` [from nVidia website](https://developer.nvidia.com/cudnn). +Unpack downloaded zip to %CUDA_PATH% (environment variable set by CUDA installer). +Alternatively, you can unpack zip to any location and set `CuDnnPath` to point to this location in `.\windows\CommonSettings.props`. +`CuDnnPath` defined in `.\windows\CommonSettings.props`. +Also, you can disable cuDNN by setting `UseCuDNN` to `false` in the property file. ### Python To build Caffe Python wrapper set `PythonSupport` to `true` in `.\windows\CommonSettings.props`. @@ -42,6 +44,11 @@ conda install --yes numpy scipy matplotlib scikit-image pip pip install protobuf ``` +#### Remark +After you have build solution with Python support, in order to use it you have to either: +1) set PythonPath environment variable to point to \Build\x64\Release\pycaffe +or +2) cp –r \Build\x64\Release\pycaffe\caffe $PYTHON_DIR\lib\site-packages ### Build Now, you should be able to build `.\windows\Caffe.sln` diff --git a/windows/CommonSettings.props.example b/windows/CommonSettings.props.example index f5b4f3a7377..5aae79fa58e 100644 --- a/windows/CommonSettings.props.example +++ b/windows/CommonSettings.props.example @@ -14,25 +14,25 @@ - compute_35,sm_35 - + Setting proper architecture is important to mimize your run and compile time. --> + compute_35,sm_35;compute_52,sm_52 - $(SolutionDir)..\..\CaffeCuDnn + $(SolutionDir)\scripts cublas.lib;cuda.lib;curand.lib;cudart.lib + cudnn.lib;$(CudaDependencies) + + $(CuDnnPath)\cuda\lib\x64;$(LibraryPath) $(CuDnnPath)\cuda\include;$(IncludePath) + $(BuildDir)\$(Platform)\$(Configuration)\ $(BuildDir)\Int\$(ProjectName)\$(Platform)\$(Configuration)\ diff --git a/windows/scripts/BinplaceCudaDependencies.cmd b/windows/scripts/BinplaceCudaDependencies.cmd index 4f5b0e480f7..330b095b11c 100644 --- a/windows/scripts/BinplaceCudaDependencies.cmd +++ b/windows/scripts/BinplaceCudaDependencies.cmd @@ -14,8 +14,13 @@ if %IS_CPU_ONLY_BUILD% == true ( copy /y "%CUDA_TOOLKIT_BIN_DIR%\curand*.dll" "%OUTPUT_DIR%" if %USE_CUDNN% == true ( - echo BinplaceCudaDependencies : Copy cunn*.dll to output. - copy /y "%CUDNN_PATH%\cuda\bin\cudnn*.dll" "%OUTPUT_DIR%" + echo BinplaceCudaDependencies : Copy cudnn*.dll to output. + + if [%CUDNN_PATH%] == [] ( + copy /y "%CUDA_TOOLKIT_BIN_DIR%\cudnn*.dll" "%OUTPUT_DIR%" + ) else ( + copy /y "%CUDNN_PATH%\cuda\bin\cudnn*.dll" "%OUTPUT_DIR%" + ) ) else ( echo BinplaceCudaDependencies : cuDNN isn't enabled. ) diff --git a/windows/scripts/PythonPostBuild.cmd b/windows/scripts/PythonPostBuild.cmd index 6eb3aa759d8..28ebcb844d7 100644 --- a/windows/scripts/PythonPostBuild.cmd +++ b/windows/scripts/PythonPostBuild.cmd @@ -5,4 +5,5 @@ echo PythonPostBuild.cmd : copy python generated scripts to output. copy /y "%SOLUTION_DIR%..\python\caffe\*.py" "%OUTPUT_DIR%pycaffe\caffe" copy /y "%SOLUTION_DIR%..\python\*.py" "%OUTPUT_DIR%pycaffe" -move /y "%OUTPUT_DIR%_caffe.*" "%OUTPUT_DIR%pycaffe\caffe" \ No newline at end of file +move /y "%OUTPUT_DIR%_caffe.*" "%OUTPUT_DIR%pycaffe\caffe" +copy /y "%OUTPUT_DIR%\*.dll" "%OUTPUT_DIR%pycaffe\caffe" \ No newline at end of file From cf7c0a4643c0d091833113b022a1630997247853 Mon Sep 17 00:00:00 2001 From: Sasa Galic Date: Mon, 7 Mar 2016 04:45:24 -0800 Subject: [PATCH 06/51] Fix issues after the latest merge --- windows/libcaffe/libcaffe.vcxproj | 5 +++++ windows/libcaffe/libcaffe.vcxproj.filters | 15 +++++++++++++++ windows/test_all/test_all.vcxproj | 1 + windows/test_all/test_all.vcxproj.filters | 3 +++ 4 files changed, 24 insertions(+) diff --git a/windows/libcaffe/libcaffe.vcxproj b/windows/libcaffe/libcaffe.vcxproj index 2a0679f2b3f..81ad1188e8c 100644 --- a/windows/libcaffe/libcaffe.vcxproj +++ b/windows/libcaffe/libcaffe.vcxproj @@ -109,6 +109,7 @@ + @@ -135,6 +136,7 @@ + @@ -208,6 +210,7 @@ + @@ -234,6 +237,7 @@ + @@ -296,6 +300,7 @@ + diff --git a/windows/libcaffe/libcaffe.vcxproj.filters b/windows/libcaffe/libcaffe.vcxproj.filters index f68cb502aef..f781b823f6b 100644 --- a/windows/libcaffe/libcaffe.vcxproj.filters +++ b/windows/libcaffe/libcaffe.vcxproj.filters @@ -165,6 +165,9 @@ src\layers + + src\layers + src\layers @@ -240,6 +243,9 @@ src\layers + + src\layers + src\layers @@ -413,6 +419,9 @@ include\layers + + include\layers + include\layers @@ -488,6 +497,9 @@ include\layers + + include\layers + include\layers @@ -727,6 +739,9 @@ cu\layers + + cu\layers + cu\layers diff --git a/windows/test_all/test_all.vcxproj b/windows/test_all/test_all.vcxproj index aa3409a8679..0988711d461 100644 --- a/windows/test_all/test_all.vcxproj +++ b/windows/test_all/test_all.vcxproj @@ -85,6 +85,7 @@ + diff --git a/windows/test_all/test_all.vcxproj.filters b/windows/test_all/test_all.vcxproj.filters index 1e2f107c956..46811c42ed0 100644 --- a/windows/test_all/test_all.vcxproj.filters +++ b/windows/test_all/test_all.vcxproj.filters @@ -47,6 +47,9 @@ src + + src + src From 3372cef6c8b5800df70ed3bc92445333a51750d0 Mon Sep 17 00:00:00 2001 From: Luis Unzueta Date: Sun, 28 Feb 2016 11:51:55 +0100 Subject: [PATCH 07/51] Add support for matcaffe build in Windows Necessary changes to enable matcaffe build in Windows: - Visual Studio project file added. - Common settings file updated. - Pre and post build scripts added. - 3rd party dependencies resolved through NuGet. - Minor code changes. --- README.md | 7 ++ cmake/Modules/FindMKL.cmake | 220 +++++++++++++++++------------------ matlab/+caffe/+test/test_solver.m | 2 +- matlab/+caffe/private/caffe_.cpp | 11 ++ windows/Caffe.sln | 11 ++ windows/CommonSettings.props.example | 15 ++- windows/matcaffe/matcaffe.def | 2 + windows/matcaffe/matcaffe.vcxproj | 128 ++++++++++++++++++++ windows/matcaffe/packages.config | 18 +++ windows/scripts/MatlabPostBuild.cmd | 9 ++ windows/scripts/MatlabPreBuild.cmd | 8 ++ 11 files changed, 319 insertions(+), 112 deletions(-) create mode 100644 windows/matcaffe/matcaffe.def create mode 100644 windows/matcaffe/matcaffe.vcxproj create mode 100644 windows/matcaffe/packages.config create mode 100644 windows/scripts/MatlabPostBuild.cmd create mode 100644 windows/scripts/MatlabPreBuild.cmd diff --git a/README.md b/README.md index 5582657a7c2..6c9caf08dcf 100644 --- a/README.md +++ b/README.md @@ -49,6 +49,13 @@ After you have build solution with Python support, in order to use it you have t 1) set PythonPath environment variable to point to \Build\x64\Release\pycaffe or 2) cp –r \Build\x64\Release\pycaffe\caffe $PYTHON_DIR\lib\site-packages + +### Matlab +To build Caffe Matlab wrapper set `MatlabSupport` to `true` and `MatlabDir` to the root of your Matlab installation in `.\windows\CommonSettings.props`. + +#### Remark +After you have build solution with Matlab support, in order to use it you have to set the generated matcaffe folder in the Matlab search path. + ### Build Now, you should be able to build `.\windows\Caffe.sln` diff --git a/cmake/Modules/FindMKL.cmake b/cmake/Modules/FindMKL.cmake index 5ab93b2d6b6..8ac6fc0c1e3 100644 --- a/cmake/Modules/FindMKL.cmake +++ b/cmake/Modules/FindMKL.cmake @@ -1,110 +1,110 @@ -# Find the MKL libraries -# -# Options: -# -# MKL_USE_SINGLE_DYNAMIC_LIBRARY : use single dynamic library interface -# MKL_USE_STATIC_LIBS : use static libraries -# MKL_MULTI_THREADED : use multi-threading -# -# This module defines the following variables: -# -# MKL_FOUND : True mkl is found -# MKL_INCLUDE_DIR : unclude directory -# MKL_LIBRARIES : the libraries to link against. - - -# ---[ Options -caffe_option(MKL_USE_SINGLE_DYNAMIC_LIBRARY "Use single dynamic library interface" ON) -caffe_option(MKL_USE_STATIC_LIBS "Use static libraries" OFF IF NOT MKL_USE_SINGLE_DYNAMIC_LIBRARY) -caffe_option(MKL_MULTI_THREADED "Use multi-threading" ON IF NOT MKL_USE_SINGLE_DYNAMIC_LIBRARY) - -# ---[ Root folders -set(INTEL_ROOT "/opt/intel" CACHE PATH "Folder contains intel libs") -find_path(MKL_ROOT include/mkl.h PATHS $ENV{MKLROOT} ${INTEL_ROOT}/mkl - DOC "Folder contains MKL") - -# ---[ Find include dir -find_path(MKL_INCLUDE_DIR mkl.h PATHS ${MKL_ROOT} PATH_SUFFIXES include) -set(__looked_for MKL_INCLUDE_DIR) - -# ---[ Find libraries -if(CMAKE_SIZEOF_VOID_P EQUAL 4) - set(__path_suffixes lib lib/ia32) -else() - set(__path_suffixes lib lib/intel64) -endif() - -set(__mkl_libs "") -if(MKL_USE_SINGLE_DYNAMIC_LIBRARY) - list(APPEND __mkl_libs rt) -else() - if(CMAKE_SIZEOF_VOID_P EQUAL 4) - if(WIN32) - list(APPEND __mkl_libs intel_c) - else() - list(APPEND __mkl_libs intel gf) - endif() - else() - list(APPEND __mkl_libs intel_lp64 gf_lp64) - endif() - - if(MKL_MULTI_THREADED) - list(APPEND __mkl_libs intel_thread) - else() - list(APPEND __mkl_libs sequential) - endif() - - list(APPEND __mkl_libs core cdft_core) -endif() - - -foreach (__lib ${__mkl_libs}) - set(__mkl_lib "mkl_${__lib}") - string(TOUPPER ${__mkl_lib} __mkl_lib_upper) - - if(MKL_USE_STATIC_LIBS) - set(__mkl_lib "lib${__mkl_lib}.a") - endif() - - find_library(${__mkl_lib_upper}_LIBRARY - NAMES ${__mkl_lib} - PATHS ${MKL_ROOT} "${MKL_INCLUDE_DIR}/.." - PATH_SUFFIXES ${__path_suffixes} - DOC "The path to Intel(R) MKL ${__mkl_lib} library") - mark_as_advanced(${__mkl_lib_upper}_LIBRARY) - - list(APPEND __looked_for ${__mkl_lib_upper}_LIBRARY) - list(APPEND MKL_LIBRARIES ${${__mkl_lib_upper}_LIBRARY}) -endforeach() - - -if(NOT MKL_USE_SINGLE_DYNAMIC_LIBRARY) - if (MKL_USE_STATIC_LIBS) - set(__iomp5_libs iomp5 libiomp5mt.lib) - else() - set(__iomp5_libs iomp5 libiomp5md.lib) - endif() - - if(WIN32) - find_path(INTEL_INCLUDE_DIR omp.h PATHS ${INTEL_ROOT} PATH_SUFFIXES include) - list(APPEND __looked_for INTEL_INCLUDE_DIR) - endif() - - find_library(MKL_RTL_LIBRARY ${__iomp5_libs} - PATHS ${INTEL_RTL_ROOT} ${INTEL_ROOT}/compiler ${MKL_ROOT}/.. ${MKL_ROOT}/../compiler - PATH_SUFFIXES ${__path_suffixes} - DOC "Path to Path to OpenMP runtime library") - - list(APPEND __looked_for MKL_RTL_LIBRARY) - list(APPEND MKL_LIBRARIES ${MKL_RTL_LIBRARY}) -endif() - - -include(FindPackageHandleStandardArgs) -find_package_handle_standard_args(MKL DEFAULT_MSG ${__looked_for}) - -if(MKL_FOUND) - message(STATUS "Found MKL (include: ${MKL_INCLUDE_DIR}, lib: ${MKL_LIBRARIES}") -endif() - -caffe_clear_vars(__looked_for __mkl_libs __path_suffixes __lib_suffix __iomp5_libs) +# Find the MKL libraries +# +# Options: +# +# MKL_USE_SINGLE_DYNAMIC_LIBRARY : use single dynamic library interface +# MKL_USE_STATIC_LIBS : use static libraries +# MKL_MULTI_THREADED : use multi-threading +# +# This module defines the following variables: +# +# MKL_FOUND : True mkl is found +# MKL_INCLUDE_DIR : unclude directory +# MKL_LIBRARIES : the libraries to link against. + + +# ---[ Options +caffe_option(MKL_USE_SINGLE_DYNAMIC_LIBRARY "Use single dynamic library interface" ON) +caffe_option(MKL_USE_STATIC_LIBS "Use static libraries" OFF IF NOT MKL_USE_SINGLE_DYNAMIC_LIBRARY) +caffe_option(MKL_MULTI_THREADED "Use multi-threading" ON IF NOT MKL_USE_SINGLE_DYNAMIC_LIBRARY) + +# ---[ Root folders +set(INTEL_ROOT "/opt/intel" CACHE PATH "Folder contains intel libs") +find_path(MKL_ROOT include/mkl.h PATHS $ENV{MKLROOT} ${INTEL_ROOT}/mkl + DOC "Folder contains MKL") + +# ---[ Find include dir +find_path(MKL_INCLUDE_DIR mkl.h PATHS ${MKL_ROOT} PATH_SUFFIXES include) +set(__looked_for MKL_INCLUDE_DIR) + +# ---[ Find libraries +if(CMAKE_SIZEOF_VOID_P EQUAL 4) + set(__path_suffixes lib lib/ia32) +else() + set(__path_suffixes lib lib/intel64) +endif() + +set(__mkl_libs "") +if(MKL_USE_SINGLE_DYNAMIC_LIBRARY) + list(APPEND __mkl_libs rt) +else() + if(CMAKE_SIZEOF_VOID_P EQUAL 4) + if(WIN32) + list(APPEND __mkl_libs intel_c) + else() + list(APPEND __mkl_libs intel gf) + endif() + else() + list(APPEND __mkl_libs intel_lp64 gf_lp64) + endif() + + if(MKL_MULTI_THREADED) + list(APPEND __mkl_libs intel_thread) + else() + list(APPEND __mkl_libs sequential) + endif() + + list(APPEND __mkl_libs core cdft_core) +endif() + + +foreach (__lib ${__mkl_libs}) + set(__mkl_lib "mkl_${__lib}") + string(TOUPPER ${__mkl_lib} __mkl_lib_upper) + + if(MKL_USE_STATIC_LIBS) + set(__mkl_lib "lib${__mkl_lib}.a") + endif() + + find_library(${__mkl_lib_upper}_LIBRARY + NAMES ${__mkl_lib} + PATHS ${MKL_ROOT} "${MKL_INCLUDE_DIR}/.." + PATH_SUFFIXES ${__path_suffixes} + DOC "The path to Intel(R) MKL ${__mkl_lib} library") + mark_as_advanced(${__mkl_lib_upper}_LIBRARY) + + list(APPEND __looked_for ${__mkl_lib_upper}_LIBRARY) + list(APPEND MKL_LIBRARIES ${${__mkl_lib_upper}_LIBRARY}) +endforeach() + + +if(NOT MKL_USE_SINGLE_DYNAMIC_LIBRARY) + if (MKL_USE_STATIC_LIBS) + set(__iomp5_libs iomp5 libiomp5mt.lib) + else() + set(__iomp5_libs iomp5 libiomp5md.lib) + endif() + + if(WIN32) + find_path(INTEL_INCLUDE_DIR omp.h PATHS ${INTEL_ROOT} PATH_SUFFIXES include) + list(APPEND __looked_for INTEL_INCLUDE_DIR) + endif() + + find_library(MKL_RTL_LIBRARY ${__iomp5_libs} + PATHS ${INTEL_RTL_ROOT} ${INTEL_ROOT}/compiler ${MKL_ROOT}/.. ${MKL_ROOT}/../compiler + PATH_SUFFIXES ${__path_suffixes} + DOC "Path to Path to OpenMP runtime library") + + list(APPEND __looked_for MKL_RTL_LIBRARY) + list(APPEND MKL_LIBRARIES ${MKL_RTL_LIBRARY}) +endif() + + +include(FindPackageHandleStandardArgs) +find_package_handle_standard_args(MKL DEFAULT_MSG ${__looked_for}) + +if(MKL_FOUND) + message(STATUS "Found MKL (include: ${MKL_INCLUDE_DIR}, lib: ${MKL_LIBRARIES}") +endif() + +caffe_clear_vars(__looked_for __mkl_libs __path_suffixes __lib_suffix __iomp5_libs) diff --git a/matlab/+caffe/+test/test_solver.m b/matlab/+caffe/+test/test_solver.m index 739258b0e85..bfd0c75f805 100644 --- a/matlab/+caffe/+test/test_solver.m +++ b/matlab/+caffe/+test/test_solver.m @@ -13,7 +13,7 @@ fid = fopen(solver_file, 'w'); fprintf(fid, [ ... - 'net: "' model_file '"\n' ... + 'net: "' strrep(model_file, '\', '/') '"\n' ... 'test_iter: 10 test_interval: 10 base_lr: 0.01 momentum: 0.9\n' ... 'weight_decay: 0.0005 lr_policy: "inv" gamma: 0.0001 power: 0.75\n' ... 'display: 100 max_iter: 100 snapshot_after_train: false\n' ]); diff --git a/matlab/+caffe/private/caffe_.cpp b/matlab/+caffe/private/caffe_.cpp index 1b1b2bff861..4f5a9827226 100644 --- a/matlab/+caffe/private/caffe_.cpp +++ b/matlab/+caffe/private/caffe_.cpp @@ -45,7 +45,14 @@ void mxCHECK_FILE_EXIST(const char* file) { static vector > > solvers_; static vector > > nets_; // init_key is generated at the beginning and everytime you call reset +#ifndef _MSC_VER // We are not using MSVC. static double init_key = static_cast(caffe_rng_rand()); +#else // We are using MSVC. +// The original statement may cause MATLAB halt on Windows when cuBLAS is used. +// Using a negative number as a flag instead of calling caffe_rng_rand(). +// init_key will be generated in entry function: mexFunction(). +static double init_key = -1; +#endif // !_MSC_VER /** ----------------------------------------------------------------- ** data conversion functions @@ -559,6 +566,10 @@ static handler_registry handlers[] = { **/ // Usage: caffe_(api_command, arg1, arg2, ...) void mexFunction(MEX_ARGS) { +#ifdef _MSC_VER + if (init_key == -1) + init_key = static_cast(caffe_rng_rand()); +#endif // _MSC_VER mexLock(); // Avoid clearing the mex file. mxCHECK(nrhs > 0, "Usage: caffe_(api_command, arg1, arg2, ...)"); // Handle input command diff --git a/windows/Caffe.sln b/windows/Caffe.sln index 9807327d263..276408f7fef 100644 --- a/windows/Caffe.sln +++ b/windows/Caffe.sln @@ -32,6 +32,11 @@ Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pycaffe", "pycaffe\pycaffe. {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} EndProjectSection EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "matcaffe", "matcaffe\matcaffe.vcxproj", "{7173D611-3A7A-4F07-943A-727C6862E8D5}" + ProjectSection(ProjectDependencies) = postProject + {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} + EndProjectSection +EndProject Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "props", "props", "{632DD6E1-28DF-42F9-AD7F-1C1F2D38765C}" ProjectSection(SolutionItems) = preProject CommonSettings.props = CommonSettings.props @@ -44,6 +49,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "scripts", "scripts", "{E2EF scripts\ProtoCompile.cmd = scripts\ProtoCompile.cmd scripts\PythonPostBuild.cmd = scripts\PythonPostBuild.cmd scripts\PythonPreBuild.cmd = scripts\PythonPreBuild.cmd + scripts\MatlabPostBuild.cmd = scripts\MatlabPostBuild.cmd + scripts\MatlabPreBuild.cmd = scripts\MatlabPreBuild.cmd EndProjectSection EndProject Global @@ -80,6 +87,10 @@ Global {38B6CE09-4B1A-4E72-A547-8A3299D8DA60}.Debug|x64.Build.0 = Debug|x64 {38B6CE09-4B1A-4E72-A547-8A3299D8DA60}.Release|x64.ActiveCfg = Release|x64 {38B6CE09-4B1A-4E72-A547-8A3299D8DA60}.Release|x64.Build.0 = Release|x64 + {7173D611-3A7A-4F07-943A-727C6862E8D5}.Debug|x64.ActiveCfg = Debug|x64 + {7173D611-3A7A-4F07-943A-727C6862E8D5}.Debug|x64.Build.0 = Debug|x64 + {7173D611-3A7A-4F07-943A-727C6862E8D5}.Release|x64.ActiveCfg = Release|x64 + {7173D611-3A7A-4F07-943A-727C6862E8D5}.Release|x64.Build.0 = Release|x64 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE diff --git a/windows/CommonSettings.props.example b/windows/CommonSettings.props.example index 5aae79fa58e..ceb9949ea1f 100644 --- a/windows/CommonSettings.props.example +++ b/windows/CommonSettings.props.example @@ -11,6 +11,9 @@ set to the root of your Python installation. If your Python installation does not contain debug libraries, debug build will not work. --> false + + false + + $(BuildDependsOn) + OriginalBuild;SkipBuild + c9666409 + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + diff --git a/windows/matcaffe/packages.config b/windows/matcaffe/packages.config new file mode 100644 index 00000000000..db022e05115 --- /dev/null +++ b/windows/matcaffe/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + diff --git a/windows/scripts/MatlabPostBuild.cmd b/windows/scripts/MatlabPostBuild.cmd new file mode 100644 index 00000000000..fac2874caba --- /dev/null +++ b/windows/scripts/MatlabPostBuild.cmd @@ -0,0 +1,9 @@ +set SOLUTION_DIR=%~1% +set OUTPUT_DIR=%~2% + +echo MatlabPostBuild.cmd : copy matlab generated scripts to output. + +@echo run_tests.m > "%temp%\excludelist.txt" +xcopy /y "%SOLUTION_DIR%..\matlab\+caffe\*.m" "%OUTPUT_DIR%matcaffe\+caffe" /exclude:%temp%\excludelist.txt +copy /y "%SOLUTION_DIR%..\matlab\+caffe\private\*.m" "%OUTPUT_DIR%matcaffe\+caffe\private" +move /y "%OUTPUT_DIR%caffe_.*" "%OUTPUT_DIR%matcaffe\+caffe\private" diff --git a/windows/scripts/MatlabPreBuild.cmd b/windows/scripts/MatlabPreBuild.cmd new file mode 100644 index 00000000000..8d1cb5ff73b --- /dev/null +++ b/windows/scripts/MatlabPreBuild.cmd @@ -0,0 +1,8 @@ +set SOLUTION_DIR=%~1% +set OUTPUT_DIR=%~2% + +echo MatlabPreBuild.cmd : Create output directories for matlab scripts. + +if not exist "%OUTPUT_DIR%\matcaffe" mkdir "%OUTPUT_DIR%\matcaffe" +if not exist "%OUTPUT_DIR%\matcaffe\+caffe" mkdir "%OUTPUT_DIR%\matcaffe\+caffe" +if not exist "%OUTPUT_DIR%\matcaffe\+caffe\private" mkdir "%OUTPUT_DIR%\matcaffe\+caffe\private" From f043ecb77609e1b7d623657c9bb8af0fdb0d4ef0 Mon Sep 17 00:00:00 2001 From: Nikola Milosavljevic Date: Thu, 10 Mar 2016 06:59:00 -0800 Subject: [PATCH 08/51] Fix readme file Include instruction to add release folder to system path for matcaffe to work. Other minor fixes. --- README.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 6c9caf08dcf..1f6abc4e887 100644 --- a/README.md +++ b/README.md @@ -45,16 +45,17 @@ pip install protobuf ``` #### Remark -After you have build solution with Python support, in order to use it you have to either: -1) set PythonPath environment variable to point to \Build\x64\Release\pycaffe -or -2) cp –r \Build\x64\Release\pycaffe\caffe $PYTHON_DIR\lib\site-packages +After you have built solution with Python support, in order to use it you have to either: +* set `PythonPath` environment variable to point to `\Build\x64\Release\pycaffe`, or +* copy folder `\Build\x64\Release\pycaffe\caffe` under `\lib\site-packages`. ### Matlab To build Caffe Matlab wrapper set `MatlabSupport` to `true` and `MatlabDir` to the root of your Matlab installation in `.\windows\CommonSettings.props`. #### Remark -After you have build solution with Matlab support, in order to use it you have to set the generated matcaffe folder in the Matlab search path. +After you have built solution with Matlab support, in order to use it you have to: +* add the generated `matcaffe` folder to Matlab search path, and +* add `\Build\x64\Release` to your system path. ### Build Now, you should be able to build `.\windows\Caffe.sln` From adc4f0c0d1e50bdb73e43e44e5d4b9a21cc3774c Mon Sep 17 00:00:00 2001 From: Pavle Josipovic Date: Fri, 11 Mar 2016 11:40:07 +0100 Subject: [PATCH 09/51] Update HDF5 and LMDB packages Installing VS on C drive is not required anymore. --- windows/caffe/caffe.vcxproj | 10 +++++----- windows/caffe/packages.config | 4 ++-- windows/compute_image_mean/compute_image_mean.vcxproj | 10 +++++----- windows/compute_image_mean/packages.config | 4 ++-- windows/convert_imageset/convert_imageset.vcxproj | 10 +++++----- windows/convert_imageset/packages.config | 4 ++-- windows/extract_features/extract_features.vcxproj | 10 +++++----- windows/extract_features/packages.config | 4 ++-- windows/libcaffe/libcaffe.vcxproj | 10 +++++----- windows/libcaffe/packages.config | 4 ++-- windows/matcaffe/matcaffe.vcxproj | 16 ++++++++-------- windows/matcaffe/packages.config | 4 ++-- windows/pycaffe/packages.config | 4 ++-- windows/pycaffe/pycaffe.vcxproj | 10 +++++----- windows/test_all/packages.config | 4 ++-- windows/test_all/test_all.vcxproj | 10 +++++----- 16 files changed, 59 insertions(+), 59 deletions(-) diff --git a/windows/caffe/caffe.vcxproj b/windows/caffe/caffe.vcxproj index 564dbbee645..214bbf91752 100644 --- a/windows/caffe/caffe.vcxproj +++ b/windows/caffe/caffe.vcxproj @@ -18,7 +18,7 @@ Win32Proj x64 caffe - e703126e + 82610725 @@ -78,7 +78,7 @@ - + @@ -90,7 +90,7 @@ - + @@ -99,7 +99,7 @@ - + @@ -113,6 +113,6 @@ - + \ No newline at end of file diff --git a/windows/caffe/packages.config b/windows/caffe/packages.config index ff68ac185a6..2f87a3da771 100644 --- a/windows/caffe/packages.config +++ b/windows/caffe/packages.config @@ -9,9 +9,9 @@ - + - + diff --git a/windows/compute_image_mean/compute_image_mean.vcxproj b/windows/compute_image_mean/compute_image_mean.vcxproj index a41ec77b614..b3a530eb4fd 100644 --- a/windows/compute_image_mean/compute_image_mean.vcxproj +++ b/windows/compute_image_mean/compute_image_mean.vcxproj @@ -18,7 +18,7 @@ Win32Proj x64 compute_image_mean - f6e60ad8 + 9b72fdf3 @@ -73,7 +73,7 @@ - + @@ -84,7 +84,7 @@ - + @@ -95,7 +95,7 @@ - + @@ -107,6 +107,6 @@ - + \ No newline at end of file diff --git a/windows/compute_image_mean/packages.config b/windows/compute_image_mean/packages.config index ff68ac185a6..2f87a3da771 100644 --- a/windows/compute_image_mean/packages.config +++ b/windows/compute_image_mean/packages.config @@ -9,9 +9,9 @@ - + - + diff --git a/windows/convert_imageset/convert_imageset.vcxproj b/windows/convert_imageset/convert_imageset.vcxproj index 7b91235de54..3927061eb98 100644 --- a/windows/convert_imageset/convert_imageset.vcxproj +++ b/windows/convert_imageset/convert_imageset.vcxproj @@ -18,7 +18,7 @@ Win32Proj x64 convert_imageset - aa5aeccc + 267c8bf4 @@ -73,7 +73,7 @@ - + @@ -84,7 +84,7 @@ - + @@ -95,7 +95,7 @@ - + @@ -107,6 +107,6 @@ - + \ No newline at end of file diff --git a/windows/convert_imageset/packages.config b/windows/convert_imageset/packages.config index ff68ac185a6..2f87a3da771 100644 --- a/windows/convert_imageset/packages.config +++ b/windows/convert_imageset/packages.config @@ -9,9 +9,9 @@ - + - + diff --git a/windows/extract_features/extract_features.vcxproj b/windows/extract_features/extract_features.vcxproj index c251edf8994..921af73cf88 100644 --- a/windows/extract_features/extract_features.vcxproj +++ b/windows/extract_features/extract_features.vcxproj @@ -18,7 +18,7 @@ Win32Proj x64 extract_features - 59a71837 + 8be3cb47 @@ -79,7 +79,7 @@ - + @@ -90,7 +90,7 @@ - + @@ -101,7 +101,7 @@ - + @@ -113,6 +113,6 @@ - + \ No newline at end of file diff --git a/windows/extract_features/packages.config b/windows/extract_features/packages.config index ff68ac185a6..2f87a3da771 100644 --- a/windows/extract_features/packages.config +++ b/windows/extract_features/packages.config @@ -9,9 +9,9 @@ - + - + diff --git a/windows/libcaffe/libcaffe.vcxproj b/windows/libcaffe/libcaffe.vcxproj index 81ad1188e8c..eead9b4d5df 100644 --- a/windows/libcaffe/libcaffe.vcxproj +++ b/windows/libcaffe/libcaffe.vcxproj @@ -36,7 +36,7 @@ - b4efcc07 + 0c91d16f @@ -358,14 +358,14 @@ - + - + @@ -374,7 +374,7 @@ - + @@ -384,6 +384,6 @@ - + \ No newline at end of file diff --git a/windows/libcaffe/packages.config b/windows/libcaffe/packages.config index ab2d5ffa952..d6588e2f0a8 100644 --- a/windows/libcaffe/packages.config +++ b/windows/libcaffe/packages.config @@ -4,9 +4,9 @@ - + - + diff --git a/windows/matcaffe/matcaffe.vcxproj b/windows/matcaffe/matcaffe.vcxproj index e7163f2f1f3..c4547a7d492 100644 --- a/windows/matcaffe/matcaffe.vcxproj +++ b/windows/matcaffe/matcaffe.vcxproj @@ -82,21 +82,21 @@ - + - - + + $(BuildDependsOn) OriginalBuild;SkipBuild - c9666409 + 5d60c5dd @@ -112,7 +112,7 @@ - + @@ -122,7 +122,7 @@ - - + + - + \ No newline at end of file diff --git a/windows/matcaffe/packages.config b/windows/matcaffe/packages.config index db022e05115..047dd90842f 100644 --- a/windows/matcaffe/packages.config +++ b/windows/matcaffe/packages.config @@ -9,9 +9,9 @@ - + - + diff --git a/windows/pycaffe/packages.config b/windows/pycaffe/packages.config index 15803452a2e..0849f7f6ed2 100644 --- a/windows/pycaffe/packages.config +++ b/windows/pycaffe/packages.config @@ -9,9 +9,9 @@ - + - + diff --git a/windows/pycaffe/pycaffe.vcxproj b/windows/pycaffe/pycaffe.vcxproj index e48a61cdb34..ea39e70006f 100644 --- a/windows/pycaffe/pycaffe.vcxproj +++ b/windows/pycaffe/pycaffe.vcxproj @@ -89,14 +89,14 @@ - - + + $(BuildDependsOn) OriginalBuild;SkipBuild - 14b5f2c8 + ce4167c6 @@ -122,8 +122,8 @@ - - + + \ No newline at end of file diff --git a/windows/test_all/packages.config b/windows/test_all/packages.config index ff68ac185a6..2f87a3da771 100644 --- a/windows/test_all/packages.config +++ b/windows/test_all/packages.config @@ -9,9 +9,9 @@ - + - + diff --git a/windows/test_all/test_all.vcxproj b/windows/test_all/test_all.vcxproj index 0988711d461..b37dd88e884 100644 --- a/windows/test_all/test_all.vcxproj +++ b/windows/test_all/test_all.vcxproj @@ -1,7 +1,7 @@ - + @@ -19,7 +19,7 @@ Win32Proj x64 test_all - f6a28848 + 1df3590e @@ -180,7 +180,7 @@ - + @@ -198,11 +198,11 @@ - + - + \ No newline at end of file From ebcd91a64eb669698dbab0719daff50887fc42a5 Mon Sep 17 00:00:00 2001 From: Sasa Galic Date: Wed, 16 Mar 2016 15:00:11 +0100 Subject: [PATCH 10/51] Fix several issues with spaces in paths Currently several scripts will fail due to paths containing spaces. Fix is to enclose paths with double quotes. --- windows/caffe/caffe.vcxproj | 4 ++-- windows/scripts/BinplaceCudaDependencies.cmd | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/windows/caffe/caffe.vcxproj b/windows/caffe/caffe.vcxproj index 214bbf91752..7f05e881422 100644 --- a/windows/caffe/caffe.vcxproj +++ b/windows/caffe/caffe.vcxproj @@ -45,7 +45,7 @@ Console - $(ScriptsDir)\FixGFlagsNaming.cmd "$(OutDir)" $(Configuration) + "$(ScriptsDir)\FixGFlagsNaming.cmd" "$(OutDir)" $(Configuration) @@ -54,7 +54,7 @@ Console - $(ScriptsDir)\FixGFlagsNaming.cmd "$(OutDir)" $(Configuration) + "$(ScriptsDir)\FixGFlagsNaming.cmd" "$(OutDir)" $(Configuration) diff --git a/windows/scripts/BinplaceCudaDependencies.cmd b/windows/scripts/BinplaceCudaDependencies.cmd index 330b095b11c..d984102882c 100644 --- a/windows/scripts/BinplaceCudaDependencies.cmd +++ b/windows/scripts/BinplaceCudaDependencies.cmd @@ -16,7 +16,7 @@ if %IS_CPU_ONLY_BUILD% == true ( if %USE_CUDNN% == true ( echo BinplaceCudaDependencies : Copy cudnn*.dll to output. - if [%CUDNN_PATH%] == [] ( + if "%CUDNN_PATH%" == "" ( copy /y "%CUDA_TOOLKIT_BIN_DIR%\cudnn*.dll" "%OUTPUT_DIR%" ) else ( copy /y "%CUDNN_PATH%\cuda\bin\cudnn*.dll" "%OUTPUT_DIR%" From 2fb47113bb4fe4b6dd9db6e1ffdcc1a9879239b4 Mon Sep 17 00:00:00 2001 From: Pavle Josipovic Date: Tue, 22 Mar 2016 03:49:28 -0700 Subject: [PATCH 11/51] Update HDF5 NuGet package Yet another update to HDF5 nuget package so that users are not required to install VS on c drive. --- windows/caffe/caffe.vcxproj | 4 ++-- windows/caffe/packages.config | 2 +- windows/compute_image_mean/compute_image_mean.vcxproj | 4 ++-- windows/compute_image_mean/packages.config | 2 +- windows/convert_imageset/convert_imageset.vcxproj | 4 ++-- windows/convert_imageset/packages.config | 2 +- windows/extract_features/extract_features.vcxproj | 4 ++-- windows/extract_features/packages.config | 2 +- windows/libcaffe/libcaffe.vcxproj | 4 ++-- windows/libcaffe/packages.config | 2 +- windows/matcaffe/matcaffe.vcxproj | 4 ++-- windows/matcaffe/packages.config | 2 +- windows/pycaffe/packages.config | 2 +- windows/pycaffe/pycaffe.vcxproj | 4 ++-- windows/test_all/packages.config | 2 +- windows/test_all/test_all.vcxproj | 4 ++-- 16 files changed, 24 insertions(+), 24 deletions(-) diff --git a/windows/caffe/caffe.vcxproj b/windows/caffe/caffe.vcxproj index 7f05e881422..2f06199784c 100644 --- a/windows/caffe/caffe.vcxproj +++ b/windows/caffe/caffe.vcxproj @@ -78,7 +78,7 @@ - + @@ -99,7 +99,7 @@ - + diff --git a/windows/caffe/packages.config b/windows/caffe/packages.config index 2f87a3da771..25a7e34d470 100644 --- a/windows/caffe/packages.config +++ b/windows/caffe/packages.config @@ -9,7 +9,7 @@ - + diff --git a/windows/compute_image_mean/compute_image_mean.vcxproj b/windows/compute_image_mean/compute_image_mean.vcxproj index b3a530eb4fd..776e88bfbb5 100644 --- a/windows/compute_image_mean/compute_image_mean.vcxproj +++ b/windows/compute_image_mean/compute_image_mean.vcxproj @@ -73,7 +73,7 @@ - + @@ -95,7 +95,7 @@ - + diff --git a/windows/compute_image_mean/packages.config b/windows/compute_image_mean/packages.config index 2f87a3da771..25a7e34d470 100644 --- a/windows/compute_image_mean/packages.config +++ b/windows/compute_image_mean/packages.config @@ -9,7 +9,7 @@ - + diff --git a/windows/convert_imageset/convert_imageset.vcxproj b/windows/convert_imageset/convert_imageset.vcxproj index 3927061eb98..4e0ab62eee4 100644 --- a/windows/convert_imageset/convert_imageset.vcxproj +++ b/windows/convert_imageset/convert_imageset.vcxproj @@ -73,7 +73,7 @@ - + @@ -95,7 +95,7 @@ - + diff --git a/windows/convert_imageset/packages.config b/windows/convert_imageset/packages.config index 2f87a3da771..25a7e34d470 100644 --- a/windows/convert_imageset/packages.config +++ b/windows/convert_imageset/packages.config @@ -9,7 +9,7 @@ - + diff --git a/windows/extract_features/extract_features.vcxproj b/windows/extract_features/extract_features.vcxproj index 921af73cf88..7233b9b2b96 100644 --- a/windows/extract_features/extract_features.vcxproj +++ b/windows/extract_features/extract_features.vcxproj @@ -79,7 +79,7 @@ - + @@ -101,7 +101,7 @@ - + diff --git a/windows/extract_features/packages.config b/windows/extract_features/packages.config index 2f87a3da771..25a7e34d470 100644 --- a/windows/extract_features/packages.config +++ b/windows/extract_features/packages.config @@ -9,7 +9,7 @@ - + diff --git a/windows/libcaffe/libcaffe.vcxproj b/windows/libcaffe/libcaffe.vcxproj index eead9b4d5df..fce0a30ed43 100644 --- a/windows/libcaffe/libcaffe.vcxproj +++ b/windows/libcaffe/libcaffe.vcxproj @@ -358,7 +358,7 @@ - + @@ -374,7 +374,7 @@ - + diff --git a/windows/libcaffe/packages.config b/windows/libcaffe/packages.config index d6588e2f0a8..3d67f16ed6c 100644 --- a/windows/libcaffe/packages.config +++ b/windows/libcaffe/packages.config @@ -4,7 +4,7 @@ - + diff --git a/windows/matcaffe/matcaffe.vcxproj b/windows/matcaffe/matcaffe.vcxproj index c4547a7d492..e127b10881f 100644 --- a/windows/matcaffe/matcaffe.vcxproj +++ b/windows/matcaffe/matcaffe.vcxproj @@ -89,7 +89,7 @@ - + @@ -122,7 +122,7 @@ - + \ No newline at end of file diff --git a/windows/matcaffe/packages.config b/windows/matcaffe/packages.config index 047dd90842f..920090a85a5 100644 --- a/windows/matcaffe/packages.config +++ b/windows/matcaffe/packages.config @@ -9,7 +9,7 @@ - + diff --git a/windows/pycaffe/packages.config b/windows/pycaffe/packages.config index 0849f7f6ed2..e0f4af8edaa 100644 --- a/windows/pycaffe/packages.config +++ b/windows/pycaffe/packages.config @@ -9,7 +9,7 @@ - + diff --git a/windows/pycaffe/pycaffe.vcxproj b/windows/pycaffe/pycaffe.vcxproj index ea39e70006f..ccf45167202 100644 --- a/windows/pycaffe/pycaffe.vcxproj +++ b/windows/pycaffe/pycaffe.vcxproj @@ -89,7 +89,7 @@ - + @@ -123,7 +123,7 @@ - + \ No newline at end of file diff --git a/windows/test_all/packages.config b/windows/test_all/packages.config index 2f87a3da771..25a7e34d470 100644 --- a/windows/test_all/packages.config +++ b/windows/test_all/packages.config @@ -9,7 +9,7 @@ - + diff --git a/windows/test_all/test_all.vcxproj b/windows/test_all/test_all.vcxproj index b37dd88e884..7761e6b86f2 100644 --- a/windows/test_all/test_all.vcxproj +++ b/windows/test_all/test_all.vcxproj @@ -1,7 +1,7 @@ - + @@ -198,7 +198,7 @@ - + From 34e8e6eb20cc39113cbe8ecd9678c03033ade3b2 Mon Sep 17 00:00:00 2001 From: Sasa Galic Date: Mon, 28 Mar 2016 15:39:31 +0200 Subject: [PATCH 12/51] Remove optimization in Appveyor build Using whole program optimization and link time code generation increases build time and makes Appveyor time out. In this checkin optimizations are removed to mitigate the Appveyor build issues. --- appveyor.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/appveyor.yml b/appveyor.yml index 235cc83dda3..a83cf9a887f 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -12,7 +12,7 @@ build_script: msbuild Caffe.sln /m /v:m /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" /p:Configuration=Debug;CpuOnlyBuild=true;UseCuDNN=false - msbuild Caffe.sln /m /v:m /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" /p:Configuration=Release;CpuOnlyBuild=true;UseCuDNN=false + msbuild Caffe.sln /m /v:m /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" /p:Configuration=Release;CpuOnlyBuild=true;UseCuDNN=false;WholeProgramOptimization=false cd .. From 9eb541ba0972726b1cdacd2504637b06a8b452e0 Mon Sep 17 00:00:00 2001 From: Sasa Galic Date: Fri, 25 Mar 2016 20:02:33 +0100 Subject: [PATCH 13/51] Support windows build for rest of the tools Currently, not all of Caffe tools can be built for Windows. In this checkin windows build for these tools is enabled. --- examples/cpp_classification/classification.cpp | 2 +- examples/mnist/convert_mnist_data.cpp | 5 + windows/Caffe.sln | 42 ++++++++ windows/classification/classification.vcxproj | 112 +++++++++++++++++++++ windows/classification/packages.config | 18 ++++ .../convert_cifar_data/convert_cifar_data.vcxproj | 112 +++++++++++++++++++++ windows/convert_cifar_data/packages.config | 18 ++++ .../convert_mnist_data/convert_mnist_data.vcxproj | 112 +++++++++++++++++++++ windows/convert_mnist_data/packages.config | 18 ++++ .../convert_mnist_siamese_data.vcxproj | 112 +++++++++++++++++++++ windows/convert_mnist_siamese_data/packages.config | 18 ++++ windows/upgrade_net_proto_binary/packages.config | 18 ++++ .../upgrade_net_proto_binary.vcxproj | 112 +++++++++++++++++++++ windows/upgrade_net_proto_text/packages.config | 18 ++++ .../upgrade_net_proto_text.vcxproj | 112 +++++++++++++++++++++ windows/upgrade_solver_proto_text/packages.config | 18 ++++ .../upgrade_solver_proto_text.vcxproj | 112 +++++++++++++++++++++ 17 files changed, 958 insertions(+), 1 deletion(-) create mode 100644 windows/classification/classification.vcxproj create mode 100644 windows/classification/packages.config create mode 100644 windows/convert_cifar_data/convert_cifar_data.vcxproj create mode 100644 windows/convert_cifar_data/packages.config create mode 100644 windows/convert_mnist_data/convert_mnist_data.vcxproj create mode 100644 windows/convert_mnist_data/packages.config create mode 100644 windows/convert_mnist_siamese_data/convert_mnist_siamese_data.vcxproj create mode 100644 windows/convert_mnist_siamese_data/packages.config create mode 100644 windows/upgrade_net_proto_binary/packages.config create mode 100644 windows/upgrade_net_proto_binary/upgrade_net_proto_binary.vcxproj create mode 100644 windows/upgrade_net_proto_text/packages.config create mode 100644 windows/upgrade_net_proto_text/upgrade_net_proto_text.vcxproj create mode 100644 windows/upgrade_solver_proto_text/packages.config create mode 100644 windows/upgrade_solver_proto_text/upgrade_solver_proto_text.vcxproj diff --git a/examples/cpp_classification/classification.cpp b/examples/cpp_classification/classification.cpp index 6b67c537a47..8affe524a89 100644 --- a/examples/cpp_classification/classification.cpp +++ b/examples/cpp_classification/classification.cpp @@ -92,7 +92,7 @@ static bool PairCompare(const std::pair& lhs, static std::vector Argmax(const std::vector& v, int N) { std::vector > pairs; for (size_t i = 0; i < v.size(); ++i) - pairs.push_back(std::make_pair(v[i], i)); + pairs.push_back(std::make_pair(v[i], static_cast(i))); std::partial_sort(pairs.begin(), pairs.begin() + N, pairs.end(), PairCompare); std::vector result; diff --git a/examples/mnist/convert_mnist_data.cpp b/examples/mnist/convert_mnist_data.cpp index 16d28093dd5..5e8c1d6c85c 100644 --- a/examples/mnist/convert_mnist_data.cpp +++ b/examples/mnist/convert_mnist_data.cpp @@ -16,6 +16,11 @@ #include #endif +#if defined(_MSC_VER) +#include +#define mkdir(X, Y) _mkdir(X) +#endif + #include #include diff --git a/windows/Caffe.sln b/windows/Caffe.sln index 276408f7fef..3a3b09d41d7 100644 --- a/windows/Caffe.sln +++ b/windows/Caffe.sln @@ -53,6 +53,20 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "scripts", "scripts", "{E2EF scripts\MatlabPreBuild.cmd = scripts\MatlabPreBuild.cmd EndProjectSection EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "convert_cifar_data", "convert_cifar_data\convert_cifar_data.vcxproj", "{B166B643-C90B-4903-B735-D2D4ED4F2248}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "classification", "classification\classification.vcxproj", "{273E7766-61AA-437C-BCA9-4CA7FE0484D4}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "convert_mnist_data", "convert_mnist_data\convert_mnist_data.vcxproj", "{73EED2A0-EED0-4514-8C95-ADA25CD3C72D}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "convert_mnist_siamese_data", "convert_mnist_siamese_data\convert_mnist_siamese_data.vcxproj", "{3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "upgrade_net_proto_binary", "upgrade_net_proto_binary\upgrade_net_proto_binary.vcxproj", "{7971DD9E-FEA9-446B-B432-F3910B8B84A8}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "upgrade_net_proto_text", "upgrade_net_proto_text\upgrade_net_proto_text.vcxproj", "{4E201A07-4464-4ECF-8D5E-6B7E3B2D896B}" +EndProject +Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "upgrade_solver_proto_text", "upgrade_solver_proto_text\upgrade_solver_proto_text.vcxproj", "{E1185C4E-1AEA-4E0E-BE85-2671E065016A}" +EndProject Global GlobalSection(SolutionConfigurationPlatforms) = preSolution Debug|x64 = Debug|x64 @@ -91,6 +105,34 @@ Global {7173D611-3A7A-4F07-943A-727C6862E8D5}.Debug|x64.Build.0 = Debug|x64 {7173D611-3A7A-4F07-943A-727C6862E8D5}.Release|x64.ActiveCfg = Release|x64 {7173D611-3A7A-4F07-943A-727C6862E8D5}.Release|x64.Build.0 = Release|x64 + {B166B643-C90B-4903-B735-D2D4ED4F2248}.Debug|x64.ActiveCfg = Debug|x64 + {B166B643-C90B-4903-B735-D2D4ED4F2248}.Debug|x64.Build.0 = Debug|x64 + {B166B643-C90B-4903-B735-D2D4ED4F2248}.Release|x64.ActiveCfg = Release|x64 + {B166B643-C90B-4903-B735-D2D4ED4F2248}.Release|x64.Build.0 = Release|x64 + {273E7766-61AA-437C-BCA9-4CA7FE0484D4}.Debug|x64.ActiveCfg = Debug|x64 + {273E7766-61AA-437C-BCA9-4CA7FE0484D4}.Debug|x64.Build.0 = Debug|x64 + {273E7766-61AA-437C-BCA9-4CA7FE0484D4}.Release|x64.ActiveCfg = Release|x64 + {273E7766-61AA-437C-BCA9-4CA7FE0484D4}.Release|x64.Build.0 = Release|x64 + {73EED2A0-EED0-4514-8C95-ADA25CD3C72D}.Debug|x64.ActiveCfg = Debug|x64 + {73EED2A0-EED0-4514-8C95-ADA25CD3C72D}.Debug|x64.Build.0 = Debug|x64 + {73EED2A0-EED0-4514-8C95-ADA25CD3C72D}.Release|x64.ActiveCfg = Release|x64 + {73EED2A0-EED0-4514-8C95-ADA25CD3C72D}.Release|x64.Build.0 = Release|x64 + {3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D}.Debug|x64.ActiveCfg = Debug|x64 + {3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D}.Debug|x64.Build.0 = Debug|x64 + {3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D}.Release|x64.ActiveCfg = Release|x64 + {3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D}.Release|x64.Build.0 = Release|x64 + {7971DD9E-FEA9-446B-B432-F3910B8B84A8}.Debug|x64.ActiveCfg = Debug|x64 + {7971DD9E-FEA9-446B-B432-F3910B8B84A8}.Debug|x64.Build.0 = Debug|x64 + {7971DD9E-FEA9-446B-B432-F3910B8B84A8}.Release|x64.ActiveCfg = Release|x64 + {7971DD9E-FEA9-446B-B432-F3910B8B84A8}.Release|x64.Build.0 = Release|x64 + {4E201A07-4464-4ECF-8D5E-6B7E3B2D896B}.Debug|x64.ActiveCfg = Debug|x64 + {4E201A07-4464-4ECF-8D5E-6B7E3B2D896B}.Debug|x64.Build.0 = Debug|x64 + {4E201A07-4464-4ECF-8D5E-6B7E3B2D896B}.Release|x64.ActiveCfg = Release|x64 + {4E201A07-4464-4ECF-8D5E-6B7E3B2D896B}.Release|x64.Build.0 = Release|x64 + {E1185C4E-1AEA-4E0E-BE85-2671E065016A}.Debug|x64.ActiveCfg = Debug|x64 + {E1185C4E-1AEA-4E0E-BE85-2671E065016A}.Debug|x64.Build.0 = Debug|x64 + {E1185C4E-1AEA-4E0E-BE85-2671E065016A}.Release|x64.ActiveCfg = Release|x64 + {E1185C4E-1AEA-4E0E-BE85-2671E065016A}.Release|x64.Build.0 = Release|x64 EndGlobalSection GlobalSection(SolutionProperties) = preSolution HideSolutionNode = FALSE diff --git a/windows/classification/classification.vcxproj b/windows/classification/classification.vcxproj new file mode 100644 index 00000000000..a607bf93a6e --- /dev/null +++ b/windows/classification/classification.vcxproj @@ -0,0 +1,112 @@ + + + + + + + + Debug + x64 + + + Release + x64 + + + + {273E7766-61AA-437C-BCA9-4CA7FE0484D4} + Win32Proj + x64 + classification + f6e60ad8 + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/classification/packages.config b/windows/classification/packages.config new file mode 100644 index 00000000000..25a7e34d470 --- /dev/null +++ b/windows/classification/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/convert_cifar_data/convert_cifar_data.vcxproj b/windows/convert_cifar_data/convert_cifar_data.vcxproj new file mode 100644 index 00000000000..90fe7d70dd4 --- /dev/null +++ b/windows/convert_cifar_data/convert_cifar_data.vcxproj @@ -0,0 +1,112 @@ + + + + + + + + Debug + x64 + + + Release + x64 + + + + {B166B643-C90B-4903-B735-D2D4ED4F2248} + Win32Proj + x64 + convert_cifar_data + f6e60ad8 + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/convert_cifar_data/packages.config b/windows/convert_cifar_data/packages.config new file mode 100644 index 00000000000..25a7e34d470 --- /dev/null +++ b/windows/convert_cifar_data/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/convert_mnist_data/convert_mnist_data.vcxproj b/windows/convert_mnist_data/convert_mnist_data.vcxproj new file mode 100644 index 00000000000..e58e7a767bf --- /dev/null +++ b/windows/convert_mnist_data/convert_mnist_data.vcxproj @@ -0,0 +1,112 @@ + + + + + + + + Debug + x64 + + + Release + x64 + + + + {73EED2A0-EED0-4514-8C95-ADA25CD3C72D} + Win32Proj + x64 + convert_mnist_data + f6e60ad8 + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/convert_mnist_data/packages.config b/windows/convert_mnist_data/packages.config new file mode 100644 index 00000000000..25a7e34d470 --- /dev/null +++ b/windows/convert_mnist_data/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/convert_mnist_siamese_data/convert_mnist_siamese_data.vcxproj b/windows/convert_mnist_siamese_data/convert_mnist_siamese_data.vcxproj new file mode 100644 index 00000000000..d437e7d0a48 --- /dev/null +++ b/windows/convert_mnist_siamese_data/convert_mnist_siamese_data.vcxproj @@ -0,0 +1,112 @@ + + + + + + + + Debug + x64 + + + Release + x64 + + + + {3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D} + Win32Proj + x64 + convert_mnist_siamese_data + f6e60ad8 + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/convert_mnist_siamese_data/packages.config b/windows/convert_mnist_siamese_data/packages.config new file mode 100644 index 00000000000..25a7e34d470 --- /dev/null +++ b/windows/convert_mnist_siamese_data/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/upgrade_net_proto_binary/packages.config b/windows/upgrade_net_proto_binary/packages.config new file mode 100644 index 00000000000..25a7e34d470 --- /dev/null +++ b/windows/upgrade_net_proto_binary/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/upgrade_net_proto_binary/upgrade_net_proto_binary.vcxproj b/windows/upgrade_net_proto_binary/upgrade_net_proto_binary.vcxproj new file mode 100644 index 00000000000..65f3b7e84f8 --- /dev/null +++ b/windows/upgrade_net_proto_binary/upgrade_net_proto_binary.vcxproj @@ -0,0 +1,112 @@ + + + + + + + + Debug + x64 + + + Release + x64 + + + + {7971DD9E-FEA9-446B-B432-F3910B8B84A8} + Win32Proj + x64 + upgrade_net_proto_binary + f6e60ad8 + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/upgrade_net_proto_text/packages.config b/windows/upgrade_net_proto_text/packages.config new file mode 100644 index 00000000000..25a7e34d470 --- /dev/null +++ b/windows/upgrade_net_proto_text/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/upgrade_net_proto_text/upgrade_net_proto_text.vcxproj b/windows/upgrade_net_proto_text/upgrade_net_proto_text.vcxproj new file mode 100644 index 00000000000..2cd46cfc5e3 --- /dev/null +++ b/windows/upgrade_net_proto_text/upgrade_net_proto_text.vcxproj @@ -0,0 +1,112 @@ + + + + + + + + Debug + x64 + + + Release + x64 + + + + {4E201A07-4464-4ECF-8D5E-6B7E3B2D896B} + Win32Proj + x64 + upgrade_net_proto_text + f6e60ad8 + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/upgrade_solver_proto_text/packages.config b/windows/upgrade_solver_proto_text/packages.config new file mode 100644 index 00000000000..25a7e34d470 --- /dev/null +++ b/windows/upgrade_solver_proto_text/packages.config @@ -0,0 +1,18 @@ + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/windows/upgrade_solver_proto_text/upgrade_solver_proto_text.vcxproj b/windows/upgrade_solver_proto_text/upgrade_solver_proto_text.vcxproj new file mode 100644 index 00000000000..239f2fbf802 --- /dev/null +++ b/windows/upgrade_solver_proto_text/upgrade_solver_proto_text.vcxproj @@ -0,0 +1,112 @@ + + + + + + + + Debug + x64 + + + Release + x64 + + + + {E1185C4E-1AEA-4E0E-BE85-2671E065016A} + Win32Proj + x64 + upgrade_solver_proto_text + f6e60ad8 + + + + Application + true + Unicode + v120 + + + Application + false + Unicode + v120 + + + + + + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) + Console + + + + + + + + {a9acef83-7b63-4574-a554-89ce869ea141} + false + true + false + true + true + + + + + + + + + + + + + + + + + + + + + + + + + + + This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file From cc46e084aa3703165f0b0e8314ebd3bee8c3c93c Mon Sep 17 00:00:00 2001 From: Sasa Galic Date: Mon, 9 May 2016 09:29:53 -0700 Subject: [PATCH 14/51] Fix issues after the latest merge --- windows/libcaffe/libcaffe.vcxproj | 2 ++ windows/libcaffe/libcaffe.vcxproj.filters | 6 ++++++ 2 files changed, 8 insertions(+) diff --git a/windows/libcaffe/libcaffe.vcxproj b/windows/libcaffe/libcaffe.vcxproj index fce0a30ed43..292a844db8a 100644 --- a/windows/libcaffe/libcaffe.vcxproj +++ b/windows/libcaffe/libcaffe.vcxproj @@ -144,6 +144,7 @@ + @@ -245,6 +246,7 @@ + diff --git a/windows/libcaffe/libcaffe.vcxproj.filters b/windows/libcaffe/libcaffe.vcxproj.filters index f781b823f6b..cbe4c60c944 100644 --- a/windows/libcaffe/libcaffe.vcxproj.filters +++ b/windows/libcaffe/libcaffe.vcxproj.filters @@ -267,6 +267,9 @@ src\layers + + src\layers + src\layers @@ -521,6 +524,9 @@ include\layers + + include\layers + include\layers From a473c7743331feaec3ee8b88646f81afff638bd0 Mon Sep 17 00:00:00 2001 From: Sasa Galic Date: Tue, 10 May 2016 02:34:21 -0700 Subject: [PATCH 15/51] Add nuget sources Appveyor seems to have issues in finding nuget packages. Adding nuget sources explicitly to mitigate CI build failures. --- windows/nuget.config | 3 +++ 1 file changed, 3 insertions(+) diff --git a/windows/nuget.config b/windows/nuget.config index fc77aae0d3f..ea7ca993c5a 100644 --- a/windows/nuget.config +++ b/windows/nuget.config @@ -1,4 +1,7 @@  + + + ..\..\NugetPackages \ No newline at end of file From 249184cacfd6f0a52c506950252ea8ba7c48655b Mon Sep 17 00:00:00 2001 From: Isaac Yang Date: Mon, 23 May 2016 15:58:55 -0700 Subject: [PATCH 16/51] Define preprocessing macro to properly report version on caffe command line option '-version'. --- windows/caffe/caffe.vcxproj | 3 +++ 1 file changed, 3 insertions(+) diff --git a/windows/caffe/caffe.vcxproj b/windows/caffe/caffe.vcxproj index 2f06199784c..d445970cc32 100644 --- a/windows/caffe/caffe.vcxproj +++ b/windows/caffe/caffe.vcxproj @@ -56,6 +56,9 @@ "$(ScriptsDir)\FixGFlagsNaming.cmd" "$(OutDir)" $(Configuration) + + NDEBUG;%(PreprocessorDefinitions);CAFFE_VERSION=1.0.0-rc3 + From 714d0acad8c66d64ddf7b83b9a239f7efc017894 Mon Sep 17 00:00:00 2001 From: Mathieu Lamarre Date: Tue, 5 Jul 2016 14:48:30 -0400 Subject: [PATCH 17/51] Remove Python.h include since Boost Python does it and also avoid adding the autolib for the debug version of Python --- python/caffe/_caffe.cpp | 2 -- 1 file changed, 2 deletions(-) diff --git a/python/caffe/_caffe.cpp b/python/caffe/_caffe.cpp index 32b5d921094..e188c4a576e 100644 --- a/python/caffe/_caffe.cpp +++ b/python/caffe/_caffe.cpp @@ -1,5 +1,3 @@ -#include // NOLINT(build/include_alpha) - // Produce deprecation warnings (needs to come before arrayobject.h inclusion). #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION From 2794c97388a0f8368c878e9b314aa6c1fb3ea412 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Wed, 12 Oct 2016 08:12:24 -0400 Subject: [PATCH 18/51] Merge cmake changes into latest windows branch --- CMakeLists.txt | 40 +++++- README.md | 134 +++++++++++++++------ appveyor.yml | 34 ++++-- cmake/ConfigGen.cmake | 44 +++++++ cmake/Cuda.cmake | 19 ++- cmake/Dependencies.cmake | 32 ++++- cmake/Modules/FindGFlags.cmake | 23 +--- cmake/Modules/FindGlog.cmake | 21 +--- cmake/Modules/FindLMDB.cmake | 8 +- cmake/Modules/FindLevelDB.cmake | 22 ++-- cmake/Modules/FindOpenBLAS.cmake | 7 +- cmake/Modules/FindSnappy.cmake | 14 ++- cmake/ProtoBuf.cmake | 8 +- cmake/Targets.cmake | 34 +++++- cmake/Templates/CaffeConfig.cmake.in | 32 ++++- cmake/Templates/export.hpp.in | 10 ++ cmake/WindowsCreateLinkHeader.cmake | 72 +++++++++++ cmake/lint.cmake | 6 +- include/caffe/common.hpp | 3 + include/caffe/layer_factory.hpp | 59 ++------- include/caffe/solver_factory.hpp | 56 ++------- python/CMakeLists.txt | 12 +- scripts/appveyor/appveyor_build_and_test.cmd | 7 ++ scripts/appveyor/appveyor_cmake_build_and_test.cmd | 72 +++++++++++ scripts/appveyor/appveyor_vs_build_and_test.cmd | 17 +++ scripts/download_prebuilt_dependencies.py | 55 +++++++++ src/caffe/CMakeLists.txt | 60 +++++++++ src/caffe/layer_factory.cpp | 73 +++++++++++ src/caffe/solver_factory.cpp | 74 ++++++++++++ src/caffe/test/CMakeLists.txt | 2 +- src/caffe/test/test_benchmark.cpp | 5 + tools/CMakeLists.txt | 5 + windows/README.md | 54 +++++++++ windows/libcaffe/libcaffe.vcxproj | 1 + windows/libcaffe/libcaffe.vcxproj.filters | 3 + 35 files changed, 901 insertions(+), 217 deletions(-) create mode 100644 cmake/Templates/export.hpp.in create mode 100644 cmake/WindowsCreateLinkHeader.cmake create mode 100644 scripts/appveyor/appveyor_build_and_test.cmd create mode 100644 scripts/appveyor/appveyor_cmake_build_and_test.cmd create mode 100644 scripts/appveyor/appveyor_vs_build_and_test.cmd create mode 100644 scripts/download_prebuilt_dependencies.py create mode 100644 src/caffe/solver_factory.cpp create mode 100644 windows/README.md diff --git a/CMakeLists.txt b/CMakeLists.txt index c5d99cef9dd..b9e9530ee90 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -24,11 +24,17 @@ include(cmake/Targets.cmake) include(cmake/Misc.cmake) include(cmake/Summary.cmake) include(cmake/ConfigGen.cmake) +include(cmake/WindowsCreateLinkHeader.cmake) # ---[ Options caffe_option(CPU_ONLY "Build Caffe without CUDA support" OFF) # TODO: rename to USE_CUDA caffe_option(USE_CUDNN "Build Caffe with cuDNN library support" ON IF NOT CPU_ONLY) -caffe_option(BUILD_SHARED_LIBS "Build shared libraries" ON) +if(MSVC) + # default to static libs + caffe_option(BUILD_SHARED_LIBS "Build shared libraries" OFF) +else() + caffe_option(BUILD_SHARED_LIBS "Build shared libraries" ON) +endif() caffe_option(BUILD_python "Build Python wrapper" ON) set(python_version "2" CACHE STRING "Specify which Python version to use") caffe_option(BUILD_matlab "Build Matlab wrapper" OFF IF UNIX OR APPLE) @@ -38,6 +44,25 @@ caffe_option(USE_OPENCV "Build with OpenCV support" ON) caffe_option(USE_LEVELDB "Build with levelDB" ON) caffe_option(USE_LMDB "Build with lmdb" ON) caffe_option(ALLOW_LMDB_NOLOCK "Allow MDB_NOLOCK when reading LMDB files (only if necessary)" OFF) +caffe_option(protobuf_MODULE_COMPATIBLE "Make the protobuf-config.cmake compatible with the module mode" ON IF MSVC) + +# CMake 3.4 introduced a WINDOWS_EXPORT_ALL_SYMBOLS target property that makes it possible to +# build shared libraries without using the usual declspec() decoration. +# See: https://blog.kitware.com/create-dlls-on-windows-without-declspec-using-new-cmake-export-all-feature/ +# and https://cmake.org/cmake/help/v3.5/prop_tgt/WINDOWS_EXPORT_ALL_SYMBOLS.html +# for details. +if(MSVC AND BUILD_SHARED_LIBS AND CMAKE_VERSION VERSION_LESS 3.4) + message(FATAL_ERROR "CMake 3.4 or newer is required to build a shared library with Microsoft Visual Studio") +endif() + +if(MSVC AND BUILD_SHARED_LIBS) + # Some tests (solver tests) fail when caffe is built as a shared library. The problem comes + # from protobuf that has a global static empty_string_ variable. Since caffe and test.testbin + # link to a static protobuf library both end up with their own instance of the empty_string_ + # variable. This causes some SEH exception to occur. In practice if the caffe executable does not link + # to protobuf this problem should not happen. Use at your own risk. + message(WARNING "Some tests (solvers) will fail when building as a shared library with MSVC") +endif() # ---[ Dependencies include(cmake/Dependencies.cmake) @@ -54,7 +79,9 @@ if(USE_libstdcpp) message("-- Warning: forcing libstdc++ (controlled by USE_libstdcpp option in cmake)") endif() -add_definitions(-DGTEST_USE_OWN_TR1_TUPLE) +if(NOT MSVC) + add_definitions(-DGTEST_USE_OWN_TR1_TUPLE) +endif() # ---[ Warnings caffe_warnings_disable(CMAKE_CXX_FLAGS -Wno-sign-compare -Wno-uninitialized) @@ -77,11 +104,16 @@ add_subdirectory(matlab) add_subdirectory(docs) # ---[ Linter target -add_custom_target(lint COMMAND ${CMAKE_COMMAND} -P ${PROJECT_SOURCE_DIR}/cmake/lint.cmake) +add_custom_target(lint COMMAND ${CMAKE_COMMAND} -DPYTHON_EXECUTABLE=${PYTHON_EXECUTABLE} -P ${PROJECT_SOURCE_DIR}/cmake/lint.cmake) # ---[ pytest target if(BUILD_python) - add_custom_target(pytest COMMAND python${python_version} -m unittest discover -s caffe/test WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/python ) + if(UNIX) + set(python_executable python${python_version}) + else() + set(python_executable ${PYTHON_EXECUTABLE}) + endif() + add_custom_target(pytest COMMAND ${python_executable} -m unittest discover -s caffe/test WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/python ) add_dependencies(pytest pycaffe) endif() diff --git a/README.md b/README.md index 1f6abc4e887..ecf54a50cb2 100644 --- a/README.md +++ b/README.md @@ -1,64 +1,120 @@ # Windows Caffe -**This is an experimental, Microsoft-led branch by Pavle Josipovic (@pavlejosipovic). It is a work-in-progress.** +**This is an experimental, communtity based branch led by Guillaume Dumont (@willyd). It is a work-in-progress.** This branch of Caffe ports the framework to Windows. [![Travis Build Status](https://api.travis-ci.org/BVLC/caffe.svg?branch=windows)](https://travis-ci.org/BVLC/caffe) Travis (Linux build) -[![Build status](https://ci.appveyor.com/api/projects/status/128eg95svel2a2xs?svg=true)] -(https://ci.appveyor.com/project/pavlejosipovic/caffe-v45qi) AppVeyor (Windows build) +[![Windows Build status](https://ci.appveyor.com/api/projects/status/lc0pdvlv89a9i9ae?svg=true)](https://ci.appveyor.com/project/willyd/caffe) AppVeyor (Windows build) ## Windows Setup -**Requirements**: Visual Studio 2013 +**Requirements**: + - Visual Studio 2013 + - CMake 3.4+ + - Python 2.7 Anaconda x64 (or Miniconda) + +you may also like to try the [ninja](https://ninja-build.org/) cmake generator as the build times can be much lower on multi-core machines. ninja can be installed easily with the `conda` package manager by adding the conda-forge channel with: +```cmd +> conda config --add channels conda-forge +> conda install ninja --yes +``` +When working with ninja you don't have the Visual Studio solutions as ninja is more akin to make. An alternative is to use [Visual Studio Code](https://code.visualstudio.com) with the CMake extensions and C++ extensions. + +### Install the caffe dependencies + +The easiest and recommended way of installing the required depedencies is by downloading the pre-built libraries using the `%CAFFE_ROOT%\scripts\download_prebuilt_dependencies.py` file. The following command should download and extract the prebuilt dependencies to your current working directory: + +```cmd +> python scripts\download_prebuilt_dependencies.py +``` + +This will create a folder called `libraries` containing all the required dependencies. Alternatively you can build them yourself by following the instructions in the [caffe-builder](https://github.com/willyd/caffe-builder) [README](https://github.com/willyd/caffe-builder/blob/master/README.md). For the remaining of these instructions we will assume that the libraries folder is in a folder defined by the `%CAFFE_DEPENDENCIES%` environment variable. + +### Build caffe + +If you are using the Ninja generator you need to setup the MSVC compiler using: +``` +> call "%VS120COMNTOOLS%..\..\VC\vcvarsall.bat" amd64 +``` +then from the caffe source folder you need to configure the cmake build +``` +> set CMAKE_GENERATOR=Ninja +> set CMAKE_CONFIGURATION=Release +> mkdir build +> cd build +> cmake -G%CMAKE_GENERATOR% -DBLAS=Open -DCMAKE_BUILD_TYPE=%CMAKE_CONFIGURATION% -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX= -C %CAFFE_DEPENDENCIES%\caffe-builder-config.cmake ..\ +> cmake --build . --config %CMAKE_CONFIGURATION% +> cmake --build . --config %CMAKE_CONFIGURATION% --target install +``` +In the above command `CMAKE_GENERATOR` can be either `Ninja` or `"Visual Studio 12 2013 Win64"` and `CMAKE_CONFIGURATION` can be `Release` or `Debug`. Please note however that Visual Studio will not parallelize the build of the CUDA files which results in much longer build times. + +In case on step in the above procedure is not working please refer to the appveyor build scripts in `%CAFFE_ROOT%\scripts\appveyor` to see the most up to date build procedure. + +### Use cuDNN -### Pre-Build Steps -Copy `.\windows\CommonSettings.props.example` to `.\windows\CommonSettings.props` +To use cuDNN you need to define the CUDNN_ROOT cache variable to point to where you unpacked the cuDNN files. For example, the build command above would become: -By defaults Windows build requires `CUDA` and `cuDNN` libraries. -Both can be disabled by adjusting build variables in `.\windows\CommonSettings.props`. -Python support is disabled by default, but can be enabled via `.\windows\CommonSettings.props` as well. -3rd party dependencies required by Caffe are automatically resolved via NuGet. +``` +> set CMAKE_GENERATOR=Ninja +> set CMAKE_CONFIGURATION=Release +> mkdir build +> cd build +> cmake -G%CMAKE_GENERATOR% -DBLAS=Open -DCMAKE_BUILD_TYPE=%CMAKE_CONFIGURATION% -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX= -DCUDNNROOT= -C %CAFFE_DEPENDENCIES%\caffe-builder-config.cmake ..\ +> cmake --build . --config %CMAKE_CONFIGURATION% +> cmake +``` +Make sure to use forward slashes (`/`) in the path. You will need to add the folder containing the cuDNN DLL to your PATH. + +### Building only for CPU + +If CUDA is not installed Caffe will default to a CPU_ONLY build. If you have CUDA installed but want a CPU only build you may use the CMake option `-DCPU_ONLY=1`. + +### Using the Python interface + +The recommended Python distribution is Anaconda or Miniconda. To successfully build the python interface you need to install the following packages: +``` +conda install --yes numpy scipy matplotlib scikit-image pip six +``` +also you will need a protobuf python package that is compatible with pre-built dependencies. This package can be installed this way: +``` +conda config --add channels willyd +conda install --yes protobuf==3.1.0.vc12 +``` +If Python is installed the default is to build the python interface and python layers. If you wish to disable the python layers or the python build use the CMake options `-DBUILD_python_layer=0` and `-DBUILD_python=0` respectively. In order to use the python interface you need to either add the `%CAFFE_ROOT%\python` folder to your python path of copy the `%CAFFE_ROOT%\python\caffe` folder to your `site_packages` folder. Also, you need to edit your `PATH` or copy the required DLLs next to the `caffe.pyd` file. Only Python 2.7 x64 has been tested on Windows. + +### Using the MATLAB interface -### CUDA -Download `CUDA Toolkit 7.5` [from nVidia website](https://developer.nvidia.com/cuda-toolkit). -If you don't have CUDA installed, you can experiment with CPU_ONLY build. -In `.\windows\CommonSettings.props` set `CpuOnlyBuild` to `true` and set `UseCuDNN` to `false`. +TODO -### cuDNN -Download `cuDNN v3` or `cuDNN v4` [from nVidia website](https://developer.nvidia.com/cudnn). -Unpack downloaded zip to %CUDA_PATH% (environment variable set by CUDA installer). -Alternatively, you can unpack zip to any location and set `CuDnnPath` to point to this location in `.\windows\CommonSettings.props`. -`CuDnnPath` defined in `.\windows\CommonSettings.props`. -Also, you can disable cuDNN by setting `UseCuDNN` to `false` in the property file. +### Building a shared library -### Python -To build Caffe Python wrapper set `PythonSupport` to `true` in `.\windows\CommonSettings.props`. -Download Miniconda 2.7 64-bit Windows installer [from Miniconda website] (http://conda.pydata.org/miniconda.html). -Install for all users and add Python to PATH (through installer). +CMake can be used to build a shared library instead of the default static library. To do so follow the above procedure and use `-DBUILD_SHARED_LIBS=ON`. Please note however, that some tests (more specifically the solver related tests) will fail since both the test exectuable and caffe library do not share static objects contained in the protobuf library. -Run the following commands from elevated command prompt: +### Running the tests or the caffe exectuable +To run the tests or any caffe exectuable you will have to update your `PATH` to include the directories where the depedencies dlls are located: ``` -conda install --yes numpy scipy matplotlib scikit-image pip -pip install protobuf +:: Prepend to avoid conflicts with other libraries with same name +> set PATH=%CAFFE_DEPENDENCIES%\bin;%CAFFE_DEPENDENCIES%\lib;%CAFFE_DEPENDENCIES%\x64\vc12\bin;%PATH% ``` +or you can use the prependpath.bat included with the prebuilt dependencies. Then the tests can be run from the build folder: +``` +cmake --build . --target runtest --config %CMAKE_CONFIGURATION% +``` + +### TODOs +- Visual Studio 2015: Prebuilt dependencies are available. Test if the build works and update appveyor config accordingly. +- Python 3.5: Create protobuf packages for 3.5. -#### Remark -After you have built solution with Python support, in order to use it you have to either: -* set `PythonPath` environment variable to point to `\Build\x64\Release\pycaffe`, or -* copy folder `\Build\x64\Release\pycaffe\caffe` under `\lib\site-packages`. +## Previous Visual Studio based build -### Matlab -To build Caffe Matlab wrapper set `MatlabSupport` to `true` and `MatlabDir` to the root of your Matlab installation in `.\windows\CommonSettings.props`. +The previous windows build based on Visual Studio project files is now deprecated. However, it is still available in the `windows` folder. Please see the [README.md](windows/README.md) in there for details. -#### Remark -After you have built solution with Matlab support, in order to use it you have to: -* add the generated `matcaffe` folder to Matlab search path, and -* add `\Build\x64\Release` to your system path. +## Known issues -### Build -Now, you should be able to build `.\windows\Caffe.sln` +- The `GPUTimer` related test cases always fail on Windows. This seems to be a difference between UNIX and Windows. +- Shared library (DLL) build will have failing tests. ## Further Details diff --git a/appveyor.yml b/appveyor.yml index a83cf9a887f..56d385899aa 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -1,19 +1,27 @@ version: 1.0.{build} clone_folder: c:\projects\caffe -build_script: -- cmd: >- - cd C:\projects\caffe\windows - - copy CommonSettings.props.example CommonSettings.props - - nuget restore Caffe.sln -PackagesDirectory ..\..\NugetPackages -ConfigFile nuget.config +environment: + matrix: + - WITH_CMAKE: 1 + CMAKE_GENERATOR: Ninja + CMAKE_CONFIG: Release + CMAKE_BUILD_SHARED_LIBS: OFF - set PATH=%PATH:nuget=hello% + - WITH_CMAKE: 1 + CMAKE_GENERATOR: Ninja + CMAKE_CONFIG: Debug + CMAKE_BUILD_SHARED_LIBS: OFF - msbuild Caffe.sln /m /v:m /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" /p:Configuration=Debug;CpuOnlyBuild=true;UseCuDNN=false + - WITH_CMAKE: 1 + CMAKE_GENERATOR: Visual Studio 12 2013 Win64 + CMAKE_CONFIG: Release + CMAKE_BUILD_SHARED_LIBS: OFF - msbuild Caffe.sln /m /v:m /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" /p:Configuration=Release;CpuOnlyBuild=true;UseCuDNN=false;WholeProgramOptimization=false + - WITH_CMAKE: 1 + CMAKE_GENERATOR: Visual Studio 12 2013 Win64 + CMAKE_CONFIG: Debug + CMAKE_BUILD_SHARED_LIBS: OFF - cd .. - - Build\x64\Release\test_all.exe --gtest_filter=-*TestTimer* \ No newline at end of file +build_script: +- cmd: >- + call scripts\appveyor\appveyor_build_and_test.cmd \ No newline at end of file diff --git a/cmake/ConfigGen.cmake b/cmake/ConfigGen.cmake index 056371110b5..fc408ffe895 100644 --- a/cmake/ConfigGen.cmake +++ b/cmake/ConfigGen.cmake @@ -56,21 +56,65 @@ function(caffe_generate_export_configs) list(APPEND Caffe_DEFINITIONS -DCPU_ONLY) endif() + # Disable autolinking on platforms that defaults to this + # e.g. Windows + list(APPEND Caffe_DEFINITIONS -DBOOST_ALL_NO_LIB) + if(USE_OPENCV) list(APPEND Caffe_DEFINITIONS -DUSE_OPENCV) endif() + set(GFLAGS_IMPORTED OFF) + foreach(_lib ${GFLAGS_LIBRARIES}) + if(TARGET ${_lib}) + set(GFLAGS_IMPORTED ON) + endif() + endforeach() + + set(GLOG_IMPORTED OFF) + foreach(_lib ${GLOG_LIBRARIES}) + if(TARGET ${_lib}) + set(GLOG_IMPORTED ON) + endif() + endforeach() + + set(HDF5_IMPORTED OFF) + foreach(_lib ${HDF5_LIBRARIES} ${HDF5_HL_LIBRARIES}) + if(TARGET ${_lib}) + set(HDF5_IMPORTED ON) + endif() + endforeach() + + set(LMDB_IMPORTED OFF) if(USE_LMDB) list(APPEND Caffe_DEFINITIONS -DUSE_LMDB) if (ALLOW_LMDB_NOLOCK) list(APPEND Caffe_DEFINITIONS -DALLOW_LMDB_NOLOCK) endif() + foreach(_lib ${LMDB_LIBRARIES}) + if(TARGET ${_lib}) + set(LMDB_IMPORTED ON) + endif() + endforeach() endif() + set(LEVELDB_IMPORTED OFF) + set(SNAPPY_IMPORTED OFF) if(USE_LEVELDB) list(APPEND Caffe_DEFINITIONS -DUSE_LEVELDB) + foreach(_lib ${LevelDB_LIBRARIES}) + if(TARGET ${_lib}) + set(LEVELDB_IMPORTED ON) + endif() + endforeach() + foreach(_lib ${Snappy_LIBRARIES}) + if(TARGET ${_lib}) + set(SNAPPY_IMPORTED ON) + endif() + endforeach() endif() + if(NOT HAVE_CUDNN) set(HAVE_CUDNN FALSE) else() diff --git a/cmake/Cuda.cmake b/cmake/Cuda.cmake index 286a42802b4..1e59071a782 100644 --- a/cmake/Cuda.cmake +++ b/cmake/Cuda.cmake @@ -36,8 +36,12 @@ function(caffe_detect_installed_gpus out_variable) ERROR_QUIET OUTPUT_STRIP_TRAILING_WHITESPACE) if(__nvcc_res EQUAL 0) + # nvcc outputs text containing line breaks when building with MSVC. + # The line below prevents CMake from inserting a variable with line + # breaks in the cache + string(REGEX MATCH "([1-9].[0-9])" __nvcc_out "${__nvcc_out}") string(REPLACE "2.1" "2.1(2.0)" __nvcc_out "${__nvcc_out}") - set(CUDA_gpu_detect_output ${__nvcc_out} CACHE INTERNAL "Returned GPU architetures from caffe_detect_gpus tool" FORCE) + set(CUDA_gpu_detect_output ${__nvcc_out} CACHE INTERNAL "Returned GPU architetures from caffe_detect_gpus tool" FORCE) endif() endif() @@ -172,13 +176,22 @@ function(detect_cuDNN) find_path(CUDNN_INCLUDE cudnn.h PATHS ${CUDNN_ROOT} $ENV{CUDNN_ROOT} ${CUDA_TOOLKIT_INCLUDE} + PATH_SUFFIXES include DOC "Path to cuDNN include directory." ) + + unset(_path_suffixes) + if(MSVC AND ${CMAKE_SIZEOF_VOID_P} EQUAL 8) + set(_path_suffixes PATH_SUFFIXES lib/x64) + else() + set(_path_suffixes PATH_SUFFIXES lib/Win32) + endif() get_filename_component(__libpath_hist ${CUDA_CUDART_LIBRARY} PATH) - find_library(CUDNN_LIBRARY NAMES libcudnn.so # libcudnn_static.a + find_library(CUDNN_LIBRARY NAMES cudnn #libcudnn.so # libcudnn_static.a PATHS ${CUDNN_ROOT} $ENV{CUDNN_ROOT} ${CUDNN_INCLUDE} ${__libpath_hist} + ${_path_suffixes} DOC "Path to cuDNN library.") - + if(CUDNN_INCLUDE AND CUDNN_LIBRARY) set(HAVE_CUDNN TRUE PARENT_SCOPE) set(CUDNN_FOUND TRUE PARENT_SCOPE) diff --git a/cmake/Dependencies.cmake b/cmake/Dependencies.cmake index c7b6a17aa69..bfa56df8505 100644 --- a/cmake/Dependencies.cmake +++ b/cmake/Dependencies.cmake @@ -4,8 +4,15 @@ set(Caffe_LINKER_LIBS "") # ---[ Boost find_package(Boost 1.46 REQUIRED COMPONENTS system thread filesystem) include_directories(SYSTEM ${Boost_INCLUDE_DIR}) +add_definitions(-DBOOST_ALL_NO_LIB) list(APPEND Caffe_LINKER_LIBS ${Boost_LIBRARIES}) +if(DEFINED MSVC) + # We should define this only when necessary, + # i.e VS 2013 Update 4 or earlier. + add_definitions(-DBOOST_NO_CXX11_TEMPLATE_ALIASES) +endif() + # ---[ Threads find_package(Threads REQUIRED) list(APPEND Caffe_LINKER_LIBS ${CMAKE_THREAD_LIBS_INIT}) @@ -24,9 +31,19 @@ list(APPEND Caffe_LINKER_LIBS ${GFLAGS_LIBRARIES}) include(cmake/ProtoBuf.cmake) # ---[ HDF5 -find_package(HDF5 COMPONENTS HL REQUIRED) +if(MSVC) + # Find HDF5 using it's hdf5-config.cmake file with MSVC + if(DEFINED HDF5_DIR) + list(APPEND CMAKE_MODULE_PATH ${HDF5_DIR}) + endif() + find_package(HDF5 COMPONENTS C HL REQUIRED) + set(HDF5_LIBRARIES hdf5-shared) + set(HDF5_HL_LIBRARIES hdf5_hl-shared) +else() + find_package(HDF5 COMPONENTS HL REQUIRED) +endif() include_directories(SYSTEM ${HDF5_INCLUDE_DIRS} ${HDF5_HL_INCLUDE_DIR}) -list(APPEND Caffe_LINKER_LIBS ${HDF5_LIBRARIES}) +list(APPEND Caffe_LINKER_LIBS ${HDF5_LIBRARIES} ${HDF5_HL_LIBRARIES}) # ---[ LMDB if(USE_LMDB) @@ -113,18 +130,18 @@ if(BUILD_python) find_package(NumPy 1.7.1) # Find the matching boost python implementation set(version ${PYTHONLIBS_VERSION_STRING}) - + STRING( REGEX REPLACE "[^0-9]" "" boost_py_version ${version} ) find_package(Boost 1.46 COMPONENTS "python-py${boost_py_version}") set(Boost_PYTHON_FOUND ${Boost_PYTHON-PY${boost_py_version}_FOUND}) - + while(NOT "${version}" STREQUAL "" AND NOT Boost_PYTHON_FOUND) STRING( REGEX REPLACE "([0-9.]+).[0-9]+" "\\1" version ${version} ) - + STRING( REGEX REPLACE "[^0-9]" "" boost_py_version ${version} ) find_package(Boost 1.46 COMPONENTS "python-py${boost_py_version}") set(Boost_PYTHON_FOUND ${Boost_PYTHON-PY${boost_py_version}_FOUND}) - + STRING( REGEX MATCHALL "([0-9.]+).[0-9]+" has_more_version ${version} ) if("${has_more_version}" STREQUAL "") break() @@ -142,6 +159,9 @@ if(BUILD_python) endif() if(PYTHONLIBS_FOUND AND NUMPY_FOUND AND Boost_PYTHON_FOUND) set(HAVE_PYTHON TRUE) + if(Boost_USE_STATIC_LIBS AND MSVC) + add_definitions(-DBOOST_PYTHON_STATIC_LIB) + endif() if(BUILD_python_layer) add_definitions(-DWITH_PYTHON_LAYER) include_directories(SYSTEM ${PYTHON_INCLUDE_DIRS} ${NUMPY_INCLUDE_DIR} ${Boost_INCLUDE_DIRS}) diff --git a/cmake/Modules/FindGFlags.cmake b/cmake/Modules/FindGFlags.cmake index 29b60f05037..f44ecc051f9 100644 --- a/cmake/Modules/FindGFlags.cmake +++ b/cmake/Modules/FindGFlags.cmake @@ -14,26 +14,15 @@ include(FindPackageHandleStandardArgs) set(GFLAGS_ROOT_DIR "" CACHE PATH "Folder contains Gflags") # We are testing only a couple of files in the include directories -if(WIN32) - find_path(GFLAGS_INCLUDE_DIR gflags/gflags.h - PATHS ${GFLAGS_ROOT_DIR}/src/windows) -else() - find_path(GFLAGS_INCLUDE_DIR gflags/gflags.h - PATHS ${GFLAGS_ROOT_DIR}) -endif() +find_path(GFLAGS_INCLUDE_DIR gflags/gflags.h + PATHS ${GFLAGS_ROOT_DIR}) if(MSVC) - find_library(GFLAGS_LIBRARY_RELEASE - NAMES libgflags - PATHS ${GFLAGS_ROOT_DIR} - PATH_SUFFIXES Release) - - find_library(GFLAGS_LIBRARY_DEBUG - NAMES libgflags-debug - PATHS ${GFLAGS_ROOT_DIR} - PATH_SUFFIXES Debug) + # rely on gflags-config.cmake + find_package(gflags NO_MODULE) - set(GFLAGS_LIBRARY optimized ${GFLAGS_LIBRARY_RELEASE} debug ${GFLAGS_LIBRARY_DEBUG}) + set(GFLAGS_LIBRARY ${gflags_LIBRARIES}) + set(GFLAGS_INCLUDE_DIR ${gflags_INCLUDE_DIRS}) else() find_library(GFLAGS_LIBRARY gflags) endif() diff --git a/cmake/Modules/FindGlog.cmake b/cmake/Modules/FindGlog.cmake index 99abbe478a0..eec263a795d 100644 --- a/cmake/Modules/FindGlog.cmake +++ b/cmake/Modules/FindGlog.cmake @@ -13,24 +13,15 @@ include(FindPackageHandleStandardArgs) set(GLOG_ROOT_DIR "" CACHE PATH "Folder contains Google glog") -if(WIN32) - find_path(GLOG_INCLUDE_DIR glog/logging.h - PATHS ${GLOG_ROOT_DIR}/src/windows) -else() - find_path(GLOG_INCLUDE_DIR glog/logging.h - PATHS ${GLOG_ROOT_DIR}) -endif() +find_path(GLOG_INCLUDE_DIR glog/logging.h + PATHS ${GLOG_ROOT_DIR}) if(MSVC) - find_library(GLOG_LIBRARY_RELEASE libglog_static - PATHS ${GLOG_ROOT_DIR} - PATH_SUFFIXES Release) - - find_library(GLOG_LIBRARY_DEBUG libglog_static - PATHS ${GLOG_ROOT_DIR} - PATH_SUFFIXES Debug) + # rely on glog-config.cmake + find_package(glog NO_MODULE) - set(GLOG_LIBRARY optimized ${GLOG_LIBRARY_RELEASE} debug ${GLOG_LIBRARY_DEBUG}) + set(GLOG_LIBRARY ${glog_LIBRARIES}) + set(GLOG_INCLUDE_DIR ${glog_INCLUDE_DIRS}) else() find_library(GLOG_LIBRARY glog PATHS ${GLOG_ROOT_DIR} diff --git a/cmake/Modules/FindLMDB.cmake b/cmake/Modules/FindLMDB.cmake index 8a817fd6f10..2f0adb1b6d6 100644 --- a/cmake/Modules/FindLMDB.cmake +++ b/cmake/Modules/FindLMDB.cmake @@ -12,8 +12,12 @@ # Copyright 2013 Conrad Steenberg # Aug 31, 2013 -find_path(LMDB_INCLUDE_DIR NAMES lmdb.h PATHS "$ENV{LMDB_DIR}/include") -find_library(LMDB_LIBRARIES NAMES lmdb PATHS "$ENV{LMDB_DIR}/lib" ) +if(MSVC) + find_package(LMDB NO_MODULE) +else() + find_path(LMDB_INCLUDE_DIR NAMES lmdb.h PATHS "$ENV{LMDB_DIR}/include") + find_library(LMDB_LIBRARIES NAMES lmdb PATHS "$ENV{LMDB_DIR}/lib" ) +endif() include(FindPackageHandleStandardArgs) find_package_handle_standard_args(LMDB DEFAULT_MSG LMDB_INCLUDE_DIR LMDB_LIBRARIES) diff --git a/cmake/Modules/FindLevelDB.cmake b/cmake/Modules/FindLevelDB.cmake index 97f08ac9349..6e6a92dd835 100644 --- a/cmake/Modules/FindLevelDB.cmake +++ b/cmake/Modules/FindLevelDB.cmake @@ -5,14 +5,20 @@ # LevelDB_FOUND - True if LevelDB found. # Look for the header file. -find_path(LevelDB_INCLUDE NAMES leveldb/db.h - PATHS $ENV{LEVELDB_ROOT}/include /opt/local/include /usr/local/include /usr/include - DOC "Path in which the file leveldb/db.h is located." ) - -# Look for the library. -find_library(LevelDB_LIBRARY NAMES leveldb - PATHS /usr/lib $ENV{LEVELDB_ROOT}/lib - DOC "Path to leveldb library." ) +if(MSVC) + find_package(LevelDB NO_MODULE) + set(LevelDB_INCLUDE ${LevelDB_INCLUDE_DIRS}) + set(LevelDB_LIBRARY ${LevelDB_LIBRARIES}) +else() + find_path(LevelDB_INCLUDE NAMES leveldb/db.h + PATHS $ENV{LEVELDB_ROOT}/include /opt/local/include /usr/local/include /usr/include + DOC "Path in which the file leveldb/db.h is located." ) + + # Look for the library. + find_library(LevelDB_LIBRARY NAMES leveldb + PATHS /usr/lib $ENV{LEVELDB_ROOT}/lib + DOC "Path to leveldb library." ) +endif() include(FindPackageHandleStandardArgs) find_package_handle_standard_args(LevelDB DEFAULT_MSG LevelDB_INCLUDE LevelDB_LIBRARY) diff --git a/cmake/Modules/FindOpenBLAS.cmake b/cmake/Modules/FindOpenBLAS.cmake index a6512ae7e4e..58e9aee8fbf 100644 --- a/cmake/Modules/FindOpenBLAS.cmake +++ b/cmake/Modules/FindOpenBLAS.cmake @@ -28,8 +28,13 @@ SET(Open_BLAS_LIB_SEARCH_PATHS $ENV{OpenBLAS_HOME}/lib ) +if(MSVC) + set(OpenBLAS_LIB_NAMES libopenblas.dll.a) +else() + set(OpenBLAS_LIB_NAMES openblas) +endif() FIND_PATH(OpenBLAS_INCLUDE_DIR NAMES cblas.h PATHS ${Open_BLAS_INCLUDE_SEARCH_PATHS}) -FIND_LIBRARY(OpenBLAS_LIB NAMES openblas PATHS ${Open_BLAS_LIB_SEARCH_PATHS}) +FIND_LIBRARY(OpenBLAS_LIB NAMES ${OpenBLAS_LIB_NAMES} PATHS ${Open_BLAS_LIB_SEARCH_PATHS}) SET(OpenBLAS_FOUND ON) diff --git a/cmake/Modules/FindSnappy.cmake b/cmake/Modules/FindSnappy.cmake index eff2a864a7b..3e4f5e6f636 100644 --- a/cmake/Modules/FindSnappy.cmake +++ b/cmake/Modules/FindSnappy.cmake @@ -7,12 +7,16 @@ # SNAPPY_FOUND # Snappy_INCLUDE_DIR # Snappy_LIBRARIES +if(MSVC) + # rely on snappy-config.cmake + find_package(Snappy NO_MODULE) +else() + find_path(Snappy_INCLUDE_DIR NAMES snappy.h + PATHS ${SNAPPY_ROOT_DIR} ${SNAPPY_ROOT_DIR}/include) -find_path(Snappy_INCLUDE_DIR NAMES snappy.h - PATHS ${SNAPPY_ROOT_DIR} ${SNAPPY_ROOT_DIR}/include) - -find_library(Snappy_LIBRARIES NAMES snappy - PATHS ${SNAPPY_ROOT_DIR} ${SNAPPY_ROOT_DIR}/lib) + find_library(Snappy_LIBRARIES NAMES snappy + PATHS ${SNAPPY_ROOT_DIR} ${SNAPPY_ROOT_DIR}/lib) +endif() include(FindPackageHandleStandardArgs) find_package_handle_standard_args(Snappy DEFAULT_MSG Snappy_INCLUDE_DIR Snappy_LIBRARIES) diff --git a/cmake/ProtoBuf.cmake b/cmake/ProtoBuf.cmake index 73f647f5fae..e1f036cb31f 100644 --- a/cmake/ProtoBuf.cmake +++ b/cmake/ProtoBuf.cmake @@ -1,7 +1,13 @@ # Finds Google Protocol Buffers library and compilers and extends # the standard cmake script with version and python generation support -find_package( Protobuf REQUIRED ) +if(MSVC) + # search using protobuf-config.cmake + find_package( Protobuf REQUIRED NO_MODULE) + set(PROTOBUF_INCLUDE_DIR ${PROTOBUF_INCLUDE_DIRS}) +else() + find_package( Protobuf REQUIRED ) +endif() include_directories(SYSTEM ${PROTOBUF_INCLUDE_DIR}) list(APPEND Caffe_LINKER_LIBS ${PROTOBUF_LIBRARIES}) diff --git a/cmake/Targets.cmake b/cmake/Targets.cmake index a796d00548f..c6bb562bdd5 100644 --- a/cmake/Targets.cmake +++ b/cmake/Targets.cmake @@ -1,7 +1,16 @@ ################################################################################################ # Defines global Caffe_LINK flag, This flag is required to prevent linker from excluding # some objects which are not addressed directly but are registered via static constructors -macro(caffe_set_caffe_link) +macro(caffe_set_caffe_link) + if(MSVC AND CMAKE_GENERATOR MATCHES Ninja) + foreach(_suffix "" ${CMAKE_CONFIGURATION_TYPES}) + if(NOT _suffix STREQUAL "") + string(TOUPPER _${_suffix} _suffix) + endif() + set(CMAKE_CXX_FLAGS${_suffix} "${CMAKE_CXX_FLAGS${_suffix}} /FS") + set(CMAKE_C_FLAGS${_suffix} "${CMAKE_C_FLAGS${_suffix}} /FS") + endforeach() + endif() if(BUILD_SHARED_LIBS) set(Caffe_LINK caffe) else() @@ -9,6 +18,8 @@ macro(caffe_set_caffe_link) set(Caffe_LINK -Wl,-force_load caffe) elseif("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU") set(Caffe_LINK -Wl,--whole-archive caffe -Wl,--no-whole-archive) + elseif(MSVC) + set(Caffe_LINK caffe) endif() endif() endmacro() @@ -52,9 +63,18 @@ endfunction() # caffe_pickup_caffe_sources() function(caffe_pickup_caffe_sources root) # put all files in source groups (visible as subfolder in many IDEs) + set(caffe_export_hdr_in ${PROJECT_SOURCE_DIR}/cmake/Templates/export.hpp.in) + set(caffe_export_hdr ${PROJECT_BINARY_DIR}/caffe/export.hpp) + set(caffe_symbols_hdr ${PROJECT_BINARY_DIR}/caffe/include_symbols.hpp) + set_source_files_properties(${caffe_export_hdr} ${caffe_symbols_hdr} PROPERTIES GENERATED TRUE) + caffe_source_group("Include" GLOB "${root}/include/caffe/*.h*") caffe_source_group("Include\\Util" GLOB "${root}/include/caffe/util/*.h*") caffe_source_group("Include" GLOB "${PROJECT_BINARY_DIR}/caffe_config.h*") + caffe_source_group("Include" GLOB "${caffe_export_hdr}") + if(MSVC AND NOT BUILD_SHARED_LIBS) + caffe_source_group("Include" GLOB "${caffe_symbols_hdr}") + endif() caffe_source_group("Source" GLOB "${root}/src/caffe/*.cpp") caffe_source_group("Source\\Util" GLOB "${root}/src/caffe/util/*.cpp") caffe_source_group("Source\\Layers" GLOB "${root}/src/caffe/layers/*.cpp") @@ -76,7 +96,13 @@ function(caffe_pickup_caffe_sources root) list(REMOVE_ITEM srcs ${test_srcs}) # adding headers to make the visible in some IDEs (Qt, VS, Xcode) - list(APPEND srcs ${hdrs} ${PROJECT_BINARY_DIR}/caffe_config.h) + list(APPEND srcs ${hdrs} + ${PROJECT_BINARY_DIR}/caffe_config.h + ${caffe_export_hdr} + ) + if(MSVC AND NOT BUILD_SHARED_LIBS) + list(APPEND srcs ${caffe_symbols_hdr}) + endif() list(APPEND test_srcs ${test_hdrs}) # collect cuda files @@ -99,6 +125,10 @@ function(caffe_pickup_caffe_sources root) set(cuda ${cuda} PARENT_SCOPE) set(test_srcs ${test_srcs} PARENT_SCOPE) set(test_cuda ${test_cuda} PARENT_SCOPE) + set(caffe_export_hdr_in ${caffe_export_hdr_in} PARENT_SCOPE) + set(caffe_export_hdr ${caffe_export_hdr} PARENT_SCOPE) + set(caffe_symbols_hdr ${caffe_symbols_hdr} PARENT_SCOPE) + endfunction() ################################################################################################ diff --git a/cmake/Templates/CaffeConfig.cmake.in b/cmake/Templates/CaffeConfig.cmake.in index 73f57ac2d74..243c9a6a70b 100644 --- a/cmake/Templates/CaffeConfig.cmake.in +++ b/cmake/Templates/CaffeConfig.cmake.in @@ -27,7 +27,13 @@ if(@USE_OPENCV@) if(EXISTS ${Caffe_OpenCV_CONFIG_PATH} AND NOT TARGET opencv_core) message(STATUS "Caffe: using OpenCV config from ${Caffe_OpenCV_CONFIG_PATH}") - include(${Caffe_OpenCV_CONFIG_PATH}/OpenCVModules.cmake) + if(MSVC) + # The path to OpenCVModules.cmake is mangled according to + # compiler and arch on Windows + include(${Caffe_OpenCV_CONFIG_PATH}/OpenConfig.cmake) + else() + include(${Caffe_OpenCV_CONFIG_PATH}/OpenCVModules.cmake) + endif() endif() else() @@ -37,6 +43,30 @@ if(@USE_OPENCV@) endif() endif() +# Handle other imported targets libraries +if(@GFLAGS_IMPORTED@) + find_package(gflags REQUIRED NO_MODULE) +endif() + +if(@GLOG_IMPORTED@) + find_package(glog REQUIRED NO_MODULE) +endif() + +if(@HDF5_IMPORTED@) + find_package(HDF5 COMPONENTS C HL REQUIRED NO_MODULE) +endif() + +if(@USE_LMDB@ AND @LMDB_IMPORTED@) + find_package(LMDB REQUIRED NO_MODULE) +endif() + +if(@USE_LEVELDB@ AND @LEVELDB_IMPORTED@) + find_package(LevelDB REQUIRED NO_MODULE) + if(@SNAPPY_IMPORTED@) + find_package(Snappy REQUIRED NO_MODULE) + endif() +endif() + # Compute paths get_filename_component(Caffe_CMAKE_DIR "${CMAKE_CURRENT_LIST_FILE}" PATH) set(Caffe_INCLUDE_DIRS "@Caffe_INCLUDE_DIRS@") diff --git a/cmake/Templates/export.hpp.in b/cmake/Templates/export.hpp.in new file mode 100644 index 00000000000..33ff222c156 --- /dev/null +++ b/cmake/Templates/export.hpp.in @@ -0,0 +1,10 @@ +#ifndef CAFFE_EXPORT_HPP_ +#define CAFFE_EXPORT_HPP_ + +// CAFFE_BUILDING_STATIC_LIB should be defined +// only by the caffe target +#if defined(_MSC_VER) && !defined(CAFFE_BUILDING_STATIC_LIB) + ${CAFFE_INCLUDE_SYMBOLS} +#endif + +#endif // CAFFE_EXPORT_HPP_ \ No newline at end of file diff --git a/cmake/WindowsCreateLinkHeader.cmake b/cmake/WindowsCreateLinkHeader.cmake new file mode 100644 index 00000000000..29e77b08953 --- /dev/null +++ b/cmake/WindowsCreateLinkHeader.cmake @@ -0,0 +1,72 @@ +set(_windows_create_link_header "${CMAKE_CURRENT_LIST_FILE}") + +# function to add a post build command to create a link header +function(windows_create_link_header target outputfile) + add_custom_command(TARGET ${target} POST_BUILD + COMMAND ${CMAKE_COMMAND} + #-DCMAKE_GENERATOR=${CMAKE_GENERATOR} + -DMSVC_VERSION=${MSVC_VERSION} + -DTARGET_FILE=$ + #-DPROJECT_BINARY_DIR=${PROJECT_BINARY_DIR} + #-DCMAKE_CURRENT_BINARY_DIR=${CMAKE_CURRENT_BINARY_DIR} + #-DCONFIGURATION=$ + -DOUTPUT_FILE=${outputfile} + -P ${_windows_create_link_header} + BYPRODUCTS ${outputfile} + ) +endfunction() + + +function(find_dumpbin var) + # MSVC_VERSION = + # 1200 = VS 6.0 + # 1300 = VS 7.0 + # 1310 = VS 7.1 + # 1400 = VS 8.0 + # 1500 = VS 9.0 + # 1600 = VS 10.0 + # 1700 = VS 11.0 + # 1800 = VS 12.0 + # 1900 = VS 14.0 + set(MSVC_PRODUCT_VERSION_1200 6.0) + set(MSVC_PRODUCT_VERSION_1300 7.0) + set(MSVC_PRODUCT_VERSION_1310 7.1) + set(MSVC_PRODUCT_VERSION_1400 8.0) + set(MSVC_PRODUCT_VERSION_1500 9.0) + set(MSVC_PRODUCT_VERSION_1600 10.0) + set(MSVC_PRODUCT_VERSION_1700 11.0) + set(MSVC_PRODUCT_VERSION_1800 12.0) + set(MSVC_PRODUCT_VERSION_1900 14.0) + get_filename_component(MSVC_VC_DIR [HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\${MSVC_PRODUCT_VERSION_${MSVC_VERSION}}\\Setup\\VC;ProductDir] REALPATH CACHE) + + find_program(DUMPBIN_EXECUTABLE dumpbin ${MSVC_VC_DIR}/bin) + if(NOT DUMPBIN_EXECUTABLE) + message(FATAL_ERROR "Could not find DUMPBIN_EXECUTABLE please define this variable") + endif() + set(${var} ${DUMPBIN_EXECUTABLE} PARENT_SCOPE) +endfunction() + +macro(print_date) + execute_process(COMMAND powershell -NoProfile -Command "get-date") +endmacro() + + +if(CMAKE_SCRIPT_MODE_FILE) + cmake_policy(SET CMP0007 NEW) + # find the dumpbin exe + find_dumpbin(dumpbin) + # execute dumpbin to generate a list of symbols + execute_process(COMMAND ${dumpbin} /SYMBOLS ${TARGET_FILE} + RESULT_VARIABLE _result + OUTPUT_VARIABLE _output + ERROR_VARIABLE _error + ) + # match all layers and solvers instantiation guard + string(REGEX MATCHALL "\\?gInstantiationGuard[^\\(\\) ]*" __symbols ${_output}) + # define a string to generate a list of pragmas + foreach(__symbol ${__symbols}) + set(__pragma "${__pragma}#pragma comment(linker, \"/include:${__symbol}\")\n") + endforeach() + file(WRITE ${OUTPUT_FILE} ${__pragma}) +endif() + diff --git a/cmake/lint.cmake b/cmake/lint.cmake index 70a006572bb..6f86937f7ed 100644 --- a/cmake/lint.cmake +++ b/cmake/lint.cmake @@ -1,6 +1,10 @@ set(CMAKE_SOURCE_DIR ..) -set(LINT_COMMAND ${CMAKE_SOURCE_DIR}/scripts/cpp_lint.py) +set(python_executable) +if(WIN32) + set(python_executable ${PYTHON_EXECUTABLE}) +endif() +set(LINT_COMMAND ${python_executable} ${CMAKE_SOURCE_DIR}/scripts/cpp_lint.py) set(SRC_FILE_EXTENSIONS h hpp hu c cpp cu cc) set(EXCLUDE_FILE_EXTENSTIONS pb.h pb.cc) set(LINT_DIRS include src/caffe examples tools python matlab) diff --git a/include/caffe/common.hpp b/include/caffe/common.hpp index 3c6a076ec2f..cd72c5f64dc 100644 --- a/include/caffe/common.hpp +++ b/include/caffe/common.hpp @@ -16,6 +16,9 @@ #include // pair #include +#ifdef CMAKE_WINDOWS_BUILD + #include "caffe/export.hpp" +#endif #include "caffe/util/device_alternate.hpp" // Convert macro to string diff --git a/include/caffe/layer_factory.hpp b/include/caffe/layer_factory.hpp index f385afccfee..be11d12016c 100644 --- a/include/caffe/layer_factory.hpp +++ b/include/caffe/layer_factory.hpp @@ -58,72 +58,31 @@ class LayerRegistry { typedef shared_ptr > (*Creator)(const LayerParameter&); typedef std::map CreatorRegistry; - static CreatorRegistry& Registry() { - static CreatorRegistry* g_registry_ = new CreatorRegistry(); - return *g_registry_; - } + static CreatorRegistry& Registry(); // Adds a creator. - static void AddCreator(const string& type, Creator creator) { - CreatorRegistry& registry = Registry(); - CHECK_EQ(registry.count(type), 0) - << "Layer type " << type << " already registered."; - registry[type] = creator; - } + static void AddCreator(const string& type, Creator creator); // Get a layer using a LayerParameter. - static shared_ptr > CreateLayer(const LayerParameter& param) { - if (Caffe::root_solver()) { - LOG(INFO) << "Creating layer " << param.name(); - } - const string& type = param.type(); - CreatorRegistry& registry = Registry(); - CHECK_EQ(registry.count(type), 1) << "Unknown layer type: " << type - << " (known types: " << LayerTypeListString() << ")"; - return registry[type](param); - } - - static vector LayerTypeList() { - CreatorRegistry& registry = Registry(); - vector layer_types; - for (typename CreatorRegistry::iterator iter = registry.begin(); - iter != registry.end(); ++iter) { - layer_types.push_back(iter->first); - } - return layer_types; - } + static shared_ptr > CreateLayer(const LayerParameter& param); + + static vector LayerTypeList(); private: // Layer registry should never be instantiated - everything is done with its // static variables. - LayerRegistry() {} - - static string LayerTypeListString() { - vector layer_types = LayerTypeList(); - string layer_types_str; - for (vector::iterator iter = layer_types.begin(); - iter != layer_types.end(); ++iter) { - if (iter != layer_types.begin()) { - layer_types_str += ", "; - } - layer_types_str += *iter; - } - return layer_types_str; - } -}; + LayerRegistry(); + static string LayerTypeListString(); +}; template class LayerRegisterer { public: LayerRegisterer(const string& type, - shared_ptr > (*creator)(const LayerParameter&)) { - // LOG(INFO) << "Registering layer type: " << type; - LayerRegistry::AddCreator(type, creator); - } + shared_ptr > (*creator)(const LayerParameter&)); }; - #define REGISTER_LAYER_CREATOR(type, creator) \ static LayerRegisterer g_creator_f_##type(#type, creator); \ static LayerRegisterer g_creator_d_##type(#type, creator) \ diff --git a/include/caffe/solver_factory.hpp b/include/caffe/solver_factory.hpp index cfff721af40..11643799139 100644 --- a/include/caffe/solver_factory.hpp +++ b/include/caffe/solver_factory.hpp @@ -56,69 +56,31 @@ class SolverRegistry { typedef Solver* (*Creator)(const SolverParameter&); typedef std::map CreatorRegistry; - static CreatorRegistry& Registry() { - static CreatorRegistry* g_registry_ = new CreatorRegistry(); - return *g_registry_; - } + static CreatorRegistry& Registry(); // Adds a creator. - static void AddCreator(const string& type, Creator creator) { - CreatorRegistry& registry = Registry(); - CHECK_EQ(registry.count(type), 0) - << "Solver type " << type << " already registered."; - registry[type] = creator; - } + static void AddCreator(const string& type, Creator creator); // Get a solver using a SolverParameter. - static Solver* CreateSolver(const SolverParameter& param) { - const string& type = param.type(); - CreatorRegistry& registry = Registry(); - CHECK_EQ(registry.count(type), 1) << "Unknown solver type: " << type - << " (known types: " << SolverTypeListString() << ")"; - return registry[type](param); - } - - static vector SolverTypeList() { - CreatorRegistry& registry = Registry(); - vector solver_types; - for (typename CreatorRegistry::iterator iter = registry.begin(); - iter != registry.end(); ++iter) { - solver_types.push_back(iter->first); - } - return solver_types; - } + static Solver* CreateSolver(const SolverParameter& param); + + static vector SolverTypeList(); private: // Solver registry should never be instantiated - everything is done with its // static variables. - SolverRegistry() {} - - static string SolverTypeListString() { - vector solver_types = SolverTypeList(); - string solver_types_str; - for (vector::iterator iter = solver_types.begin(); - iter != solver_types.end(); ++iter) { - if (iter != solver_types.begin()) { - solver_types_str += ", "; - } - solver_types_str += *iter; - } - return solver_types_str; - } -}; + SolverRegistry(); // {} + static string SolverTypeListString(); +}; template class SolverRegisterer { public: SolverRegisterer(const string& type, - Solver* (*creator)(const SolverParameter&)) { - // LOG(INFO) << "Registering solver type: " << type; - SolverRegistry::AddCreator(type, creator); - } + Solver* (*creator)(const SolverParameter&)); }; - #define REGISTER_SOLVER_CREATOR(type, creator) \ static SolverRegisterer g_creator_f_##type(#type, creator); \ static SolverRegisterer g_creator_d_##type(#type, creator) \ diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt index a22641401f0..b97067cb6be 100644 --- a/python/CMakeLists.txt +++ b/python/CMakeLists.txt @@ -6,9 +6,12 @@ endif() include_directories(${PYTHON_INCLUDE_DIRS} ${NUMPY_INCLUDE_DIR} ${Boost_INCLUDE_DIRS}) file(GLOB_RECURSE python_srcs ${PROJECT_SOURCE_DIR}/python/*.cpp) -add_library(pycaffe SHARED ${python_srcs}) +add_library(pycaffe MODULE ${python_srcs}) target_link_libraries(pycaffe ${Caffe_LINK} ${PYTHON_LIBRARIES} ${Boost_LIBRARIES}) set_target_properties(pycaffe PROPERTIES PREFIX "" OUTPUT_NAME "_caffe") +if(MSVC) + set_target_properties(pycaffe PROPERTIES SUFFIX ".pyd") +endif() caffe_default_properties(pycaffe) if(UNIX OR APPLE) @@ -19,6 +22,13 @@ if(UNIX OR APPLE) COMMAND touch ${PROJECT_SOURCE_DIR}/python/caffe/proto/__init__.py COMMAND cp ${proto_gen_folder}/*.py ${PROJECT_SOURCE_DIR}/python/caffe/proto/ COMMENT "Creating symlink ${__linkname} -> ${PROJECT_BINARY_DIR}/lib/_caffe${Caffe_POSTFIX}.so") +elseif(WIN32) + add_custom_command(TARGET pycaffe POST_BUILD + COMMAND ${CMAKE_COMMAND} -E copy_if_different $ ${PROJECT_SOURCE_DIR}/python/caffe + COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_SOURCE_DIR}/python/caffe/proto + COMMAND ${CMAKE_COMMAND} -E touch ${PROJECT_SOURCE_DIR}/python/caffe/proto/__init__.py + COMMAND (robocopy "\"${proto_gen_folder}\" \"${PROJECT_SOURCE_DIR}/python/caffe/proto\" *.py") ^& IF %ERRORLEVEL% LEQ 4 exit /B 0 + COMMENT "Creating symlink ${__linkname} -> ${PROJECT_BINARY_DIR}/lib/_caffe.pyd") endif() # ---[ Install diff --git a/scripts/appveyor/appveyor_build_and_test.cmd b/scripts/appveyor/appveyor_build_and_test.cmd new file mode 100644 index 00000000000..19c7cf41b6f --- /dev/null +++ b/scripts/appveyor/appveyor_build_and_test.cmd @@ -0,0 +1,7 @@ +if "%WITH_CMAKE%" == "1" ( + echo "Building with CMake" + call %~dp0appveyor_cmake_build_and_test.cmd +) else ( + echo "Building with Visual Studio" + call %~dp0appveyor_vs_build_and_test.cmd +) diff --git a/scripts/appveyor/appveyor_cmake_build_and_test.cmd b/scripts/appveyor/appveyor_cmake_build_and_test.cmd new file mode 100644 index 00000000000..1b973862a59 --- /dev/null +++ b/scripts/appveyor/appveyor_cmake_build_and_test.cmd @@ -0,0 +1,72 @@ +@echo off + +:: Set python 2.7 with conda as the default python +set PATH=C:\Miniconda-x64;C:\Miniconda-x64\Scripts;C:\Miniconda-x64\Library\bin;%PATH% +:: Check that we have the right python version +python --version +:: Add the required channels +conda config --add channels conda-forge +conda config --add channels willyd +:: Update conda +conda update conda -y +:: Create an environment +conda install --yes cmake ninja numpy scipy protobuf==3.1.0.vc12 six scikit-image + +:: Create build directory and configure cmake +mkdir build +pushd build +:: Download dependencies from VS 2013 x64 +python ..\scripts\download_prebuilt_dependencies.py --msvc_version v120 +:: Add the dependencies to the PATH +:: Prepending is crucial since the hdf5 dll may conflict with python's +call %cd%\libraries\prependpath.bat +:: Setup the environement for VS 2013 x64 +call "%VS120COMNTOOLS%..\..\VC\vcvarsall.bat" amd64 +:: Configure using cmake and using the caffe-builder dependencies +cmake -G"%CMAKE_GENERATOR%" ^ + -DBLAS=Open ^ + -DCMAKE_BUILD_TYPE=%CMAKE_CONFIG% ^ + -DBUILD_SHARED_LIBS=%CMAKE_BUILD_SHARED_LIBS% ^ + -C libraries\caffe-builder-config.cmake ^ + ..\ + +:: Build the library and tools +cmake --build . --config %CMAKE_CONFIG% + +if ERRORLEVEL 1 ( + echo Build failed + exit /b 1 +) + +:: Build and exectute the tests +if "%CMAKE_BUILD_SHARED_LIBS%"=="OFF" ( + :: Run the tests only for static lib as the shared lib is causing an issue. + cmake --build . --target runtest --config %CMAKE_CONFIG% + + if ERRORLEVEL 1 ( + echo Tests failed + exit /b 1 + ) + + :: Run python tests only in Release build since + :: the _caffe module is _caffe-d is debug + if "%CMAKE_CONFIG%"=="Release" ( + :: Run the python tests + cmake --build . --target pytest + + if ERRORLEVEL 1 ( + echo Python tests failed + exit /b 1 + ) + ) +) + +:: Lint +cmake --build . --target lint --config %CMAKE_CONFIG% + +if ERRORLEVEL 1 ( + echo Lint failed + exit /b 1 +) + +popd \ No newline at end of file diff --git a/scripts/appveyor/appveyor_vs_build_and_test.cmd b/scripts/appveyor/appveyor_vs_build_and_test.cmd new file mode 100644 index 00000000000..72194a7aa86 --- /dev/null +++ b/scripts/appveyor/appveyor_vs_build_and_test.cmd @@ -0,0 +1,17 @@ +@echo off + +cd C:\projects\caffe\windows + +copy CommonSettings.props.example CommonSettings.props + +nuget restore Caffe.sln -PackagesDirectory ..\..\NugetPackages -ConfigFile nuget.config + +set PATH=%PATH:nuget=hello% + +msbuild Caffe.sln /m /v:m /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" /p:Configuration=Debug;CpuOnlyBuild=true;UseCuDNN=false + +msbuild Caffe.sln /m /v:m /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" /p:Configuration=Release;CpuOnlyBuild=true;UseCuDNN=false;WholeProgramOptimization=false + +cd .. + +Build\x64\Release\test_all.exe --gtest_filter=-*TestTimer* \ No newline at end of file diff --git a/scripts/download_prebuilt_dependencies.py b/scripts/download_prebuilt_dependencies.py new file mode 100644 index 00000000000..618532301b7 --- /dev/null +++ b/scripts/download_prebuilt_dependencies.py @@ -0,0 +1,55 @@ +#!/usr/bin/python +# +# copyright Guillaume Dumont (2016) + +import os +import sys +import urllib +import hashlib +import argparse +import tarfile + +from download_model_binary import reporthook + +WIN_DEPENDENCIES_URLS = dict( + v120=("https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v120_x64_py27_1.0.1.tar.bz2", + "3f45fe3f27b27a7809f9de1bd85e56888b01dbe2"), + v140=("https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v140_x64_py27_1.0.1.tar.bz2", + "427faf33745cf8cd70c7d043c85db7dda7243122"), +) + +# function for checking SHA1. +def model_checks_out(filename, sha1): + with open(filename, 'rb') as f: + return hashlib.sha1(f.read()).hexdigest() == sha1 + + +if __name__ == '__main__': + parser = argparse.ArgumentParser( + description='Download prebuilt dependencies for windows.') + parser.add_argument('--msvc_version', default='v120', choices=['v120', 'v140']) + args = parser.parse_args() + + # get the appropriate url + try: + url, sha1 = WIN_DEPENDENCIES_URLS[args.msvc_version] + except KeyError: + print('ERROR: Could not find url for MSVC version = {}.'.format(args.msvc_version)) + sys.exit(1) + + dep_filename = os.path.split(url)[1] + # Download binaries + print("Downloading dependencies. Please wait...") + urllib.urlretrieve(url, dep_filename, reporthook) + if not model_checks_out(dep_filename, sha1): + print('ERROR: dependencies did not download correctly! Run this again.') + sys.exit(1) + print("\nDone.") + + # Extract the binaries from the tar file + tar = tarfile.open(dep_filename, 'r:bz2') + print("Extracting dependencies. Please wait...") + tar.extractall() + print("Done.") + tar.close() + diff --git a/src/caffe/CMakeLists.txt b/src/caffe/CMakeLists.txt index 8a80c940488..7514d634394 100644 --- a/src/caffe/CMakeLists.txt +++ b/src/caffe/CMakeLists.txt @@ -12,8 +12,41 @@ caffe_default_properties(proto) # creates 'test_srcs', 'srcs', 'test_cuda', 'cuda' lists caffe_pickup_caffe_sources(${PROJECT_SOURCE_DIR}) +# add this option here since CUDA will not honor +# target_compile_definitions +if(MSVC AND NOT BUILD_SHARED_LIBS) + set(_caffe_static_compile_def -DCAFFE_BUILDING_STATIC_LIB) +endif() + if(HAVE_CUDA) + # collect any compile definitions from imported targets. This important so that + # preprocessor macros such as GLOG_NO_ABBREVIATED_SEVERITIES are defined. + # this is required since CUDA macros do not honor the INTERFACE_COMPILE_DEFINITIONS + unset(__cuda_options) + foreach(__lib ${Caffe_LINKER_LIBS}) + if(TARGET ${__lib}) + get_target_property(__interface_compile_definitions ${__lib} INTERFACE_COMPILE_DEFINITIONS) + if(__interface_compile_definitions) + foreach(__def ${__interface_compile_definitions}) + # espace any parentheses because they are failing the build + # see cmake issue https://cmake.org/Bug/view.php?id=16065 + string(REPLACE "(" "\\\(" __def_escaped ${__def}) + string(REPLACE ")" "\\\)" __def_escaped ${__def_escaped}) + # add the required -D flag + list(APPEND __cuda_options "-D${__def_escaped}") + endforeach() + endif() + endif() + endforeach() + list(APPEND __cuda_options ${_caffe_static_compile_def}) + # add the required definitions + add_definitions(${__cuda_options}) + # it seems that using the OPTIONS argument like: + # caffe_cuda_compile(cuda_objs ${cuda} OPTIONS ${__cuda_options}) + # does not work. Use add/remove_definitions instead. caffe_cuda_compile(cuda_objs ${cuda}) + # remove them + remove_definitions(${__cuda_options}) list(APPEND srcs ${cuda_objs} ${cuda}) endif() @@ -24,6 +57,30 @@ set_target_properties(caffe PROPERTIES VERSION ${CAFFE_TARGET_VERSION} SOVERSION ${CAFFE_TARGET_SOVERSION} ) +if(MSVC AND BUILD_SHARED_LIBS) + # CMake 3.4 introduced a WINDOWS_EXPORT_ALL_SYMBOLS target property that makes it possible to + # build shared libraries without using the usual declspec() decoration. + # See: https://blog.kitware.com/create-dlls-on-windows-without-declspec-using-new-cmake-export-all-feature/ + # and https://cmake.org/cmake/help/v3.5/prop_tgt/WINDOWS_EXPORT_ALL_SYMBOLS.html + # for details. + set_target_properties(caffe PROPERTIES WINDOWS_EXPORT_ALL_SYMBOLS TRUE) +elseif(MSVC AND NOT BUILD_SHARED_LIBS) + # add a custom build command that generates a list of symbols + # to force linking. This is required because MSVC as nothing + # the whole-archive option + windows_create_link_header(caffe ${caffe_symbols_hdr}) + get_filename_component(_name ${caffe_symbols_hdr} NAME) + set(CAFFE_INCLUDE_SYMBOLS "#include \"caffe/${_name}\"") + # definition needed to include CMake generated files + target_compile_definitions(caffe PRIVATE ${_caffe_static_compile_def} + PUBLIC -DCMAKE_WINDOWS_BUILD) +endif() +if(MSVC) + # Disable Boost autolinking for consuming projects + target_compile_definitions(caffe PUBLIC -DBOOST_ALL_NO_LIB) +endif() + +configure_file(${caffe_export_hdr_in} ${caffe_export_hdr}) # ---[ Tests add_subdirectory(test) @@ -32,6 +89,9 @@ set_target_properties(caffe PROPERTIES install(DIRECTORY ${Caffe_INCLUDE_DIR}/caffe DESTINATION include) install(FILES ${proto_hdrs} DESTINATION include/caffe/proto) install(TARGETS caffe proto EXPORT CaffeTargets DESTINATION lib) +if(MSVC AND NOT BUILD_SHARED_LIBS) + install(FILES ${caffe_export_hdr} ${caffe_symbols_hdr} DESTINATION include/caffe) +endif() file(WRITE ${PROJECT_BINARY_DIR}/__init__.py) list(APPEND proto_python ${PROJECT_BINARY_DIR}/__init__.py) diff --git a/src/caffe/layer_factory.cpp b/src/caffe/layer_factory.cpp index e967bd6181c..41d6bf5e312 100644 --- a/src/caffe/layer_factory.cpp +++ b/src/caffe/layer_factory.cpp @@ -4,6 +4,7 @@ #include #endif #include +#include #include "caffe/layer.hpp" #include "caffe/layer_factory.hpp" @@ -33,6 +34,78 @@ namespace caffe { +template +typename LayerRegistry::CreatorRegistry& +LayerRegistry::Registry() { + static CreatorRegistry* g_registry_ = new CreatorRegistry(); + return *g_registry_; +} + +// Adds a creator. +template +void LayerRegistry::AddCreator(const string& type, Creator creator) { + CreatorRegistry& registry = Registry(); + CHECK_EQ(registry.count(type), 0) << "Layer type " << type + << " already registered."; + registry[type] = creator; +} + +// Get a layer using a LayerParameter. +template +shared_ptr > LayerRegistry::CreateLayer( + const LayerParameter& param) { + if (Caffe::root_solver()) { + LOG(INFO) << "Creating layer " << param.name(); + } + const string& type = param.type(); + CreatorRegistry& registry = Registry(); + CHECK_EQ(registry.count(type), 1) + << "Unknown layer type: " << type + << " (known types: " << LayerTypeListString() << ")"; + return registry[type](param); +} + +template +vector LayerRegistry::LayerTypeList() { + CreatorRegistry& registry = Registry(); + vector layer_types; + for (typename CreatorRegistry::iterator iter = registry.begin(); + iter != registry.end(); ++iter) { + layer_types.push_back(iter->first); + } + return layer_types; +} + +// Layer registry should never be instantiated - everything is done with its +// static variables. +template +LayerRegistry::LayerRegistry() {} + +template +string LayerRegistry::LayerTypeListString() { + vector layer_types = LayerTypeList(); + string layer_types_str; + for (vector::iterator iter = layer_types.begin(); + iter != layer_types.end(); ++iter) { + if (iter != layer_types.begin()) { + layer_types_str += ", "; + } + layer_types_str += *iter; + } + return layer_types_str; +} + +template +LayerRegisterer::LayerRegisterer( + const string& type, + shared_ptr > (*creator)(const LayerParameter&)) { + // LOG(INFO) << "Registering layer type: " << type; + LayerRegistry::AddCreator(type, creator); +} + +INSTANTIATE_CLASS(LayerRegistry); +INSTANTIATE_CLASS(LayerRegisterer); + // Get convolution layer according to engine. template shared_ptr > GetConvolutionLayer( diff --git a/src/caffe/solver_factory.cpp b/src/caffe/solver_factory.cpp new file mode 100644 index 00000000000..8ee74546fd7 --- /dev/null +++ b/src/caffe/solver_factory.cpp @@ -0,0 +1,74 @@ +#include +#include + +#include "caffe/solver_factory.hpp" + +namespace caffe { + +template +typename SolverRegistry::CreatorRegistry& +SolverRegistry::Registry() { + static CreatorRegistry* g_registry_ = new CreatorRegistry; + return *g_registry_; +} + +template +void SolverRegistry::AddCreator(const string& type, Creator creator) { + CreatorRegistry& registry = Registry(); + CHECK_EQ(registry.count(type), 0) << "Solver type " << type + << " already registered."; + registry[type] = creator; +} + +// Get a solver using a SolverParameter. +template +Solver* SolverRegistry::CreateSolver( + const SolverParameter& param) { + const string& type = param.type(); + CreatorRegistry& registry = Registry(); + CHECK_EQ(registry.count(type), 1) + << "Unknown solver type: " << type + << " (known types: " << SolverTypeListString() << ")"; + return registry[type](param); +} + +template +vector SolverRegistry::SolverTypeList() { + CreatorRegistry& registry = Registry(); + vector solver_types; + for (typename CreatorRegistry::iterator iter = registry.begin(); + iter != registry.end(); ++iter) { + solver_types.push_back(iter->first); + } + return solver_types; +} + +// Solver registry should never be instantiated - everything is done with its +// static variables. +template +SolverRegistry::SolverRegistry() {} + +template +string SolverRegistry::SolverTypeListString() { + vector solver_types = SolverTypeList(); + string solver_types_str; + for (vector::iterator iter = solver_types.begin(); + iter != solver_types.end(); ++iter) { + if (iter != solver_types.begin()) { + solver_types_str += ", "; + } + solver_types_str += *iter; + } + return solver_types_str; +} + +template +SolverRegisterer::SolverRegisterer( + const string& type, Solver* (*creator)(const SolverParameter&)) { + SolverRegistry::AddCreator(type, creator); +} + +INSTANTIATE_CLASS(SolverRegistry); +INSTANTIATE_CLASS(SolverRegisterer); + +} // namespace caffe diff --git a/src/caffe/test/CMakeLists.txt b/src/caffe/test/CMakeLists.txt index 35a803f2f41..1cc9f507ed6 100644 --- a/src/caffe/test/CMakeLists.txt +++ b/src/caffe/test/CMakeLists.txt @@ -33,4 +33,4 @@ caffe_set_runtime_directory(${the_target} "${PROJECT_BINARY_DIR}/test") # ---[ Adding runtest add_custom_target(runtest COMMAND ${the_target} ${test_args} - WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}) + WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}) \ No newline at end of file diff --git a/src/caffe/test/test_benchmark.cpp b/src/caffe/test/test_benchmark.cpp index b03fdf69a8a..96cd1bd0ba9 100644 --- a/src/caffe/test/test_benchmark.cpp +++ b/src/caffe/test/test_benchmark.cpp @@ -9,7 +9,12 @@ namespace caffe { +#ifdef _MSC_VER +// Timer tests have issues on appveyor +const float kMillisecondsThreshold = 50; +#else const float kMillisecondsThreshold = 30; +#endif template class BenchmarkTest : public MultiDeviceTest {}; diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt index 02fbd5cadd8..197d31a746b 100644 --- a/tools/CMakeLists.txt +++ b/tools/CMakeLists.txt @@ -22,6 +22,11 @@ foreach(source ${srcs}) # restore output name without suffix if(name MATCHES "caffe.bin") set_target_properties(${name} PROPERTIES OUTPUT_NAME caffe) + if(MSVC) + # the exectuable will have an import library with the same name as the caffe lib + # so change the import library to avoid name clashes + set_target_properties(${name} PROPERTIES IMPORT_SUFFIX ".bin.lib") + endif() endif() # Install diff --git a/windows/README.md b/windows/README.md new file mode 100644 index 00000000000..6b94121c940 --- /dev/null +++ b/windows/README.md @@ -0,0 +1,54 @@ +# Windows Caffe + +This is the old Visual Studio based build of caffe. The procedure below was left here for reference and may not work. This build will be removed in the near future in favor of the CMake based build. + +## Windows Setup +**Requirements**: Visual Studio 2013 + +### Pre-Build Steps +Copy `.\windows\CommonSettings.props.example` to `.\windows\CommonSettings.props` + +By defaults Windows build requires `CUDA` and `cuDNN` libraries. +Both can be disabled by adjusting build variables in `.\windows\CommonSettings.props`. +Python support is disabled by default, but can be enabled via `.\windows\CommonSettings.props` as well. +3rd party dependencies required by Caffe are automatically resolved via NuGet. + +### CUDA +Download `CUDA Toolkit 7.5` [from nVidia website](https://developer.nvidia.com/cuda-toolkit). +If you don't have CUDA installed, you can experiment with CPU_ONLY build. +In `.\windows\CommonSettings.props` set `CpuOnlyBuild` to `true` and set `UseCuDNN` to `false`. + +### cuDNN +Download `cuDNN v3` or `cuDNN v4` [from nVidia website](https://developer.nvidia.com/cudnn). +Unpack downloaded zip to %CUDA_PATH% (environment variable set by CUDA installer). +Alternatively, you can unpack zip to any location and set `CuDnnPath` to point to this location in `.\windows\CommonSettings.props`. +`CuDnnPath` defined in `.\windows\CommonSettings.props`. +Also, you can disable cuDNN by setting `UseCuDNN` to `false` in the property file. + +### Python +To build Caffe Python wrapper set `PythonSupport` to `true` in `.\windows\CommonSettings.props`. +Download Miniconda 2.7 64-bit Windows installer [from Miniconda website] (http://conda.pydata.org/miniconda.html). +Install for all users and add Python to PATH (through installer). + +Run the following commands from elevated command prompt: + +``` +conda install --yes numpy scipy matplotlib scikit-image pip +pip install protobuf +``` + +#### Remark +After you have built solution with Python support, in order to use it you have to either: +* set `PythonPath` environment variable to point to `\Build\x64\Release\pycaffe`, or +* copy folder `\Build\x64\Release\pycaffe\caffe` under `\lib\site-packages`. + +### Matlab +To build Caffe Matlab wrapper set `MatlabSupport` to `true` and `MatlabDir` to the root of your Matlab installation in `.\windows\CommonSettings.props`. + +#### Remark +After you have built solution with Matlab support, in order to use it you have to: +* add the generated `matcaffe` folder to Matlab search path, and +* add `\Build\x64\Release` to your system path. + +### Build +Now, you should be able to build `.\windows\Caffe.sln` \ No newline at end of file diff --git a/windows/libcaffe/libcaffe.vcxproj b/windows/libcaffe/libcaffe.vcxproj index 292a844db8a..139ccedb202 100644 --- a/windows/libcaffe/libcaffe.vcxproj +++ b/windows/libcaffe/libcaffe.vcxproj @@ -175,6 +175,7 @@ + diff --git a/windows/libcaffe/libcaffe.vcxproj.filters b/windows/libcaffe/libcaffe.vcxproj.filters index cbe4c60c944..0a7244d49f5 100644 --- a/windows/libcaffe/libcaffe.vcxproj.filters +++ b/windows/libcaffe/libcaffe.vcxproj.filters @@ -336,6 +336,9 @@ src\util + + src + From 630cbc183ca24a54910a9587c572c97de701a30f Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Mon, 17 Oct 2016 22:37:33 -0400 Subject: [PATCH 19/51] Updated appveyor badge for windows branch --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ecf54a50cb2..feee734bb15 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ This branch of Caffe ports the framework to Windows. [![Travis Build Status](https://api.travis-ci.org/BVLC/caffe.svg?branch=windows)](https://travis-ci.org/BVLC/caffe) Travis (Linux build) -[![Windows Build status](https://ci.appveyor.com/api/projects/status/lc0pdvlv89a9i9ae?svg=true)](https://ci.appveyor.com/project/willyd/caffe) AppVeyor (Windows build) +[![Windows Build status](https://ci.appveyor.com/api/projects/status/6xpwyq0y9ffdj9pb/branch/windows?svg=true)](https://ci.appveyor.com/project/willyd/caffe-4pvka/branch/windows) AppVeyor (Windows build) ## Windows Setup **Requirements**: From e90123e012414a96dc92242d167c8aa6b866b643 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Wed, 19 Oct 2016 12:26:39 -0400 Subject: [PATCH 20/51] Make download prebuilt dependencies compatible with python 3.5 --- scripts/download_prebuilt_dependencies.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/scripts/download_prebuilt_dependencies.py b/scripts/download_prebuilt_dependencies.py index 618532301b7..be5397ba71e 100644 --- a/scripts/download_prebuilt_dependencies.py +++ b/scripts/download_prebuilt_dependencies.py @@ -4,11 +4,11 @@ import os import sys -import urllib import hashlib import argparse import tarfile +from six.moves import urllib from download_model_binary import reporthook WIN_DEPENDENCIES_URLS = dict( @@ -40,7 +40,7 @@ def model_checks_out(filename, sha1): dep_filename = os.path.split(url)[1] # Download binaries print("Downloading dependencies. Please wait...") - urllib.urlretrieve(url, dep_filename, reporthook) + urllib.request.urlretrieve(url, dep_filename, reporthook) if not model_checks_out(dep_filename, sha1): print('ERROR: dependencies did not download correctly! Run this again.') sys.exit(1) From 6f451f51cff52b0143fa90c7c84efa0d1350f330 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Wed, 19 Oct 2016 09:59:02 -0400 Subject: [PATCH 21/51] Added CI for VS 2015 and workaround for VS 2015 bug --- appveyor.yml | 27 ++++++++++++++++++++++ python/caffe/_caffe.cpp | 26 +++++++++++++++++++++ scripts/appveyor/appveyor_cmake_build_and_test.cmd | 12 ++++++---- 3 files changed, 61 insertions(+), 4 deletions(-) diff --git a/appveyor.yml b/appveyor.yml index 56d385899aa..f44227ed713 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -3,21 +3,48 @@ clone_folder: c:\projects\caffe environment: matrix: - WITH_CMAKE: 1 + MSVC_VERSION: 14 CMAKE_GENERATOR: Ninja CMAKE_CONFIG: Release CMAKE_BUILD_SHARED_LIBS: OFF - WITH_CMAKE: 1 + MSVC_VERSION: 14 CMAKE_GENERATOR: Ninja CMAKE_CONFIG: Debug CMAKE_BUILD_SHARED_LIBS: OFF - WITH_CMAKE: 1 + MSVC_VERSION: 14 + CMAKE_GENERATOR: Visual Studio 14 2015 Win64 + CMAKE_CONFIG: Release + CMAKE_BUILD_SHARED_LIBS: OFF + + - WITH_CMAKE: 1 + MSVC_VERSION: 14 + CMAKE_GENERATOR: Visual Studio 14 2015 Win64 + CMAKE_CONFIG: Debug + CMAKE_BUILD_SHARED_LIBS: OFF + - WITH_CMAKE: 1 + MSVC_VERSION: 12 + CMAKE_GENERATOR: Ninja + CMAKE_CONFIG: Release + CMAKE_BUILD_SHARED_LIBS: OFF + + - WITH_CMAKE: 1 + MSVC_VERSION: 12 + CMAKE_GENERATOR: Ninja + CMAKE_CONFIG: Debug + CMAKE_BUILD_SHARED_LIBS: OFF + + - WITH_CMAKE: 1 + MSVC_VERSION: 12 CMAKE_GENERATOR: Visual Studio 12 2013 Win64 CMAKE_CONFIG: Release CMAKE_BUILD_SHARED_LIBS: OFF - WITH_CMAKE: 1 + MSVC_VERSION: 12 CMAKE_GENERATOR: Visual Studio 12 2013 Win64 CMAKE_CONFIG: Debug CMAKE_BUILD_SHARED_LIBS: OFF diff --git a/python/caffe/_caffe.cpp b/python/caffe/_caffe.cpp index 695a5f41cac..9c98c339de4 100644 --- a/python/caffe/_caffe.cpp +++ b/python/caffe/_caffe.cpp @@ -37,6 +37,32 @@ } \ } while (0) +#if defined(_MSC_VER) && (_MSC_FULL_VER >= 190024210) +// Workaround for VS 2015 Update 3 which breaks boost python +// See: http://stackoverflow.com/questions/38261530/unresolved-external-symbols-since-visual-studio-2015-update-3-boost-python-link +// and https://msdn.microsoft.com/vs-knownissues/vs2015-update3 +#define BP_GET_POINTER(cls, dtype) \ +namespace boost { \ +template <> \ +caffe::cls const volatile * \ +get_pointer const volatile >( \ + class caffe::cls const volatile *c) { \ + return c; \ +} \ +} + +BP_GET_POINTER(Net, float); +BP_GET_POINTER(Layer, float); +BP_GET_POINTER(Solver, float); +BP_GET_POINTER(SGDSolver, float); +BP_GET_POINTER(NesterovSolver, float); +BP_GET_POINTER(AdaGradSolver, float); +BP_GET_POINTER(RMSPropSolver, float); +BP_GET_POINTER(AdaDeltaSolver, float); +BP_GET_POINTER(AdamSolver, float); + +#endif + namespace bp = boost::python; namespace caffe { diff --git a/scripts/appveyor/appveyor_cmake_build_and_test.cmd b/scripts/appveyor/appveyor_cmake_build_and_test.cmd index 1b973862a59..b5307af6e01 100644 --- a/scripts/appveyor/appveyor_cmake_build_and_test.cmd +++ b/scripts/appveyor/appveyor_cmake_build_and_test.cmd @@ -10,18 +10,22 @@ conda config --add channels willyd :: Update conda conda update conda -y :: Create an environment +:: Todo create protobuf package for vc14 conda install --yes cmake ninja numpy scipy protobuf==3.1.0.vc12 six scikit-image :: Create build directory and configure cmake mkdir build pushd build -:: Download dependencies from VS 2013 x64 -python ..\scripts\download_prebuilt_dependencies.py --msvc_version v120 +:: Download dependencies from VS x64 +python ..\scripts\download_prebuilt_dependencies.py --msvc_version v%MSVC_VERSION%0 :: Add the dependencies to the PATH :: Prepending is crucial since the hdf5 dll may conflict with python's call %cd%\libraries\prependpath.bat -:: Setup the environement for VS 2013 x64 -call "%VS120COMNTOOLS%..\..\VC\vcvarsall.bat" amd64 +:: Setup the environement for VS x64 +@setlocal EnableDelayedExpansion +set batch_file=!VS%MSVC_VERSION%0COMNTOOLS!..\..\VC\vcvarsall.bat +@endlocal & set batch_file=%batch_file% +call "%batch_file%" amd64 :: Configure using cmake and using the caffe-builder dependencies cmake -G"%CMAKE_GENERATOR%" ^ -DBLAS=Open ^ From f19a8a94c2d767648801daa4d7fca13aebfd0c40 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Tue, 1 Nov 2016 09:25:08 -0400 Subject: [PATCH 22/51] Updated README --- README.md | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index feee734bb15..6b07af58077 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,11 @@ This branch of Caffe ports the framework to Windows. ## Windows Setup **Requirements**: - - Visual Studio 2013 + - Visual Studio 2013 or 2015 - CMake 3.4+ - Python 2.7 Anaconda x64 (or Miniconda) + - CUDA 7.5 or 8.0 (optional) (use CUDA 8 if using Visual Studio 2015) + - cuDNN v5 (optional) you may also like to try the [ninja](https://ninja-build.org/) cmake generator as the build times can be much lower on multi-core machines. ninja can be installed easily with the `conda` package manager by adding the conda-forge channel with: ```cmd @@ -23,10 +25,13 @@ When working with ninja you don't have the Visual Studio solutions as ninja is m ### Install the caffe dependencies -The easiest and recommended way of installing the required depedencies is by downloading the pre-built libraries using the `%CAFFE_ROOT%\scripts\download_prebuilt_dependencies.py` file. The following command should download and extract the prebuilt dependencies to your current working directory: +The easiest and recommended way of installing the required depedencies is by downloading the pre-built libraries using the `%CAFFE_ROOT%\scripts\download_prebuilt_dependencies.py` file. Depending on your compiler one of the following commands should download and extract the prebuilt dependencies to your current working directory: ```cmd -> python scripts\download_prebuilt_dependencies.py +:: Install Visual Studio 2013 dependencies +> python scripts\download_prebuilt_dependencies.py --msvc_version=v120 +:: Or install Visual Studio 2015 dependencies +> python scripts\download_prebuilt_dependencies.py --msvc_version=v140 ``` This will create a folder called `libraries` containing all the required dependencies. Alternatively you can build them yourself by following the instructions in the [caffe-builder](https://github.com/willyd/caffe-builder) [README](https://github.com/willyd/caffe-builder/blob/master/README.md). For the remaining of these instructions we will assume that the libraries folder is in a folder defined by the `%CAFFE_DEPENDENCIES%` environment variable. @@ -47,9 +52,9 @@ then from the caffe source folder you need to configure the cmake build > cmake --build . --config %CMAKE_CONFIGURATION% > cmake --build . --config %CMAKE_CONFIGURATION% --target install ``` -In the above command `CMAKE_GENERATOR` can be either `Ninja` or `"Visual Studio 12 2013 Win64"` and `CMAKE_CONFIGURATION` can be `Release` or `Debug`. Please note however that Visual Studio will not parallelize the build of the CUDA files which results in much longer build times. +In the above command `CMAKE_GENERATOR` can be either `Ninja`, `"Visual Studio 12 2013 Win64"` or `"Visual Studio 14 2015 Win64"` and `CMAKE_CONFIGURATION` can be `Release` or `Debug`. Please note however that Visual Studio will not parallelize the build of the CUDA files which results in much longer build times. -In case on step in the above procedure is not working please refer to the appveyor build scripts in `%CAFFE_ROOT%\scripts\appveyor` to see the most up to date build procedure. +In case one of the steps in the above procedure is not working please refer to the appveyor build scripts in `%CAFFE_ROOT%\scripts\appveyor` to see the most up to date build procedure. ### Use cuDNN @@ -96,7 +101,10 @@ CMake can be used to build a shared library instead of the default static librar To run the tests or any caffe exectuable you will have to update your `PATH` to include the directories where the depedencies dlls are located: ``` :: Prepend to avoid conflicts with other libraries with same name +:: For VS 2013 > set PATH=%CAFFE_DEPENDENCIES%\bin;%CAFFE_DEPENDENCIES%\lib;%CAFFE_DEPENDENCIES%\x64\vc12\bin;%PATH% +:: For VS 2015 +> set PATH=%CAFFE_DEPENDENCIES%\bin;%CAFFE_DEPENDENCIES%\lib;%CAFFE_DEPENDENCIES%\x64\vc14\bin;%PATH% ``` or you can use the prependpath.bat included with the prebuilt dependencies. Then the tests can be run from the build folder: ``` @@ -104,8 +112,7 @@ cmake --build . --target runtest --config %CMAKE_CONFIGURATION% ``` ### TODOs -- Visual Studio 2015: Prebuilt dependencies are available. Test if the build works and update appveyor config accordingly. -- Python 3.5: Create protobuf packages for 3.5. +- Python 3.5: Create protobuf packages for 3.5. Rebuild dependencies especially boost python with 3.5. ## Previous Visual Studio based build From 5f29ad49bdb09b3bae4dadcba8fe07c52b56e81a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mikael=20Lepist=C3=B6?= Date: Sun, 6 Nov 2016 19:04:25 +0200 Subject: [PATCH 23/51] Clarified CUDNN install instructions Fixed `CUDNN_ROOT` to be written correctly. Added missing pieces to alst cmake install command Added example which part of unpacked cudnn.zip CUDNN_ROOT path should point --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 6b07af58077..72ee1e26a9f 100644 --- a/README.md +++ b/README.md @@ -58,16 +58,16 @@ In case one of the steps in the above procedure is not working please refer to t ### Use cuDNN -To use cuDNN you need to define the CUDNN_ROOT cache variable to point to where you unpacked the cuDNN files. For example, the build command above would become: +To use cuDNN you need to define the CUDNN_ROOT cache variable to point to where you unpacked the cuDNN files e.g. `C:/Users/myuser/Projects/machine-learning/cudnn-8.0-windows10-x64-v5.1/cuda`. For example, the build command above would become: ``` > set CMAKE_GENERATOR=Ninja > set CMAKE_CONFIGURATION=Release > mkdir build > cd build -> cmake -G%CMAKE_GENERATOR% -DBLAS=Open -DCMAKE_BUILD_TYPE=%CMAKE_CONFIGURATION% -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX= -DCUDNNROOT= -C %CAFFE_DEPENDENCIES%\caffe-builder-config.cmake ..\ +> cmake -G%CMAKE_GENERATOR% -DBLAS=Open -DCMAKE_BUILD_TYPE=%CMAKE_CONFIGURATION% -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX= -DCUDNN_ROOT=C:/Users/myuser/Projects/machine-learning/cudnn-8.0-windows10-x64-v5.1/cuda -C %CAFFE_DEPENDENCIES%\caffe-builder-config.cmake ..\ > cmake --build . --config %CMAKE_CONFIGURATION% -> cmake +> cmake --build . --config %CMAKE_CONFIGURATION% --target install ``` Make sure to use forward slashes (`/`) in the path. You will need to add the folder containing the cuDNN DLL to your PATH. From a77e8d87260297a13c04de4122f391e022a523a2 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Mon, 7 Nov 2016 17:04:07 -0500 Subject: [PATCH 24/51] Added build script and updated README.md accordingly. Added initial support for python 3.5 --- CMakeLists.txt | 17 +- README.md | 104 +++++------ appveyor.yml | 59 +++--- scripts/appveyor/appveyor_build_and_test.cmd | 7 - scripts/appveyor/appveyor_cmake_build_and_test.cmd | 76 -------- scripts/appveyor/appveyor_vs_build_and_test.cmd | 17 -- scripts/build_win.cmd | 206 +++++++++++++++++++++ scripts/download_prebuilt_dependencies.py | 23 ++- 8 files changed, 304 insertions(+), 205 deletions(-) delete mode 100644 scripts/appveyor/appveyor_build_and_test.cmd delete mode 100644 scripts/appveyor/appveyor_cmake_build_and_test.cmd delete mode 100644 scripts/appveyor/appveyor_vs_build_and_test.cmd create mode 100644 scripts/build_win.cmd diff --git a/CMakeLists.txt b/CMakeLists.txt index 98a1bb75fa0..dd8cfdb8c89 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,4 +1,12 @@ cmake_minimum_required(VERSION 2.8.7) +if(MSVC) + # CMake 3.4 introduced a WINDOWS_EXPORT_ALL_SYMBOLS target property that makes it possible to + # build shared libraries without using the usual declspec() decoration. + # See: https://blog.kitware.com/create-dlls-on-windows-without-declspec-using-new-cmake-export-all-feature/ + # and https://cmake.org/cmake/help/v3.5/prop_tgt/WINDOWS_EXPORT_ALL_SYMBOLS.html + # for details. + cmake_minimum_required(VERSION 3.4) +endif() if(POLICY CMP0046) cmake_policy(SET CMP0046 NEW) endif() @@ -46,15 +54,6 @@ caffe_option(USE_LMDB "Build with lmdb" ON) caffe_option(ALLOW_LMDB_NOLOCK "Allow MDB_NOLOCK when reading LMDB files (only if necessary)" OFF) caffe_option(protobuf_MODULE_COMPATIBLE "Make the protobuf-config.cmake compatible with the module mode" ON IF MSVC) -# CMake 3.4 introduced a WINDOWS_EXPORT_ALL_SYMBOLS target property that makes it possible to -# build shared libraries without using the usual declspec() decoration. -# See: https://blog.kitware.com/create-dlls-on-windows-without-declspec-using-new-cmake-export-all-feature/ -# and https://cmake.org/cmake/help/v3.5/prop_tgt/WINDOWS_EXPORT_ALL_SYMBOLS.html -# for details. -if(MSVC AND BUILD_SHARED_LIBS AND CMAKE_VERSION VERSION_LESS 3.4) - message(FATAL_ERROR "CMake 3.4 or newer is required to build a shared library with Microsoft Visual Studio") -endif() - if(MSVC AND BUILD_SHARED_LIBS) # Some tests (solver tests) fail when caffe is built as a shared library. The problem comes # from protobuf that has a global static empty_string_ variable. Since caffe and test.testbin diff --git a/README.md b/README.md index 72ee1e26a9f..58b6e2aed16 100644 --- a/README.md +++ b/README.md @@ -9,23 +9,40 @@ This branch of Caffe ports the framework to Windows. [![Windows Build status](https://ci.appveyor.com/api/projects/status/6xpwyq0y9ffdj9pb/branch/windows?svg=true)](https://ci.appveyor.com/project/willyd/caffe-4pvka/branch/windows) AppVeyor (Windows build) ## Windows Setup -**Requirements**: + +### Requirements + - Visual Studio 2013 or 2015 - - CMake 3.4+ - - Python 2.7 Anaconda x64 (or Miniconda) + - [CMake](https://cmake.org/) 3.4 or higher (Visual Studio and [Ninja](https://ninja-build.org/) generators are supported) + - Python 2.7 Anaconda x64 (or Miniconda). - CUDA 7.5 or 8.0 (optional) (use CUDA 8 if using Visual Studio 2015) - cuDNN v5 (optional) -you may also like to try the [ninja](https://ninja-build.org/) cmake generator as the build times can be much lower on multi-core machines. ninja can be installed easily with the `conda` package manager by adding the conda-forge channel with: + We assume that `cmake.exe` and `python.exe` are on your `PATH`. + +### Configuring and Building Caffe + +The fastest method to get started with caffe on Windows is by executing the following commands in a `cmd` prompt (we use `C:\Projects` as a root folder for the remainder of the instructions): ```cmd -> conda config --add channels conda-forge -> conda install ninja --yes +C:\Projects> git clone https://github.com/BVLC/caffe.git +C:\Projects> cd caffe +C:\Projects\caffe> git checkout windows +:: Edit any of the options inside build_win.cmd to suit your needs +C:\Projects\caffe> scripts\build_win.cmd ``` -When working with ninja you don't have the Visual Studio solutions as ninja is more akin to make. An alternative is to use [Visual Studio Code](https://code.visualstudio.com) with the CMake extensions and C++ extensions. +The `build_win.cmd` script should be executed once to download the dependencies, create the Visual Studio project files (or the ninja build files) and build the Release configuration. After that you should add the required folders to your `PATH` by executing the following command: +```cmd +C:\Projects\caffe> call build\libraries\prependpath.bat +``` +Once this is done you can use the `pycaffe` interface or run `caffe.exe` from the command line. If you want to debug the `caffe.exe` exectuable, open Visual Studio from a `cmd.exe` prompt that has the required directories in its `PATH` variable and open the `C:\Projects\caffe\build\Caffe.sln` and proceed as normal. Alternatively, you can copy the required DLLs next to the `caffe.exe` ( or `caffe-d.exe` in Debug). + +Should you encounter any error please post the output of the above commands by redirecting the output to a file and open a topic on the [caffe-users list](https://groups.google.com/forum/#!forum/caffe-users) mailing list. + +Below is a more complete description of some of the steps involved in building caffe. ### Install the caffe dependencies -The easiest and recommended way of installing the required depedencies is by downloading the pre-built libraries using the `%CAFFE_ROOT%\scripts\download_prebuilt_dependencies.py` file. Depending on your compiler one of the following commands should download and extract the prebuilt dependencies to your current working directory: +The easiest and recommended way of installing the required dependencies is by downloading the pre-built libraries using the [scripts\download_prebuilt_dependencies.py](scripts\download_prebuilt_dependencies.py) file. Depending on your compiler one of the following commands should download and extract the prebuilt dependencies to your current working directory: ```cmd :: Install Visual Studio 2013 dependencies @@ -36,40 +53,24 @@ The easiest and recommended way of installing the required depedencies is by dow This will create a folder called `libraries` containing all the required dependencies. Alternatively you can build them yourself by following the instructions in the [caffe-builder](https://github.com/willyd/caffe-builder) [README](https://github.com/willyd/caffe-builder/blob/master/README.md). For the remaining of these instructions we will assume that the libraries folder is in a folder defined by the `%CAFFE_DEPENDENCIES%` environment variable. -### Build caffe - -If you are using the Ninja generator you need to setup the MSVC compiler using: -``` -> call "%VS120COMNTOOLS%..\..\VC\vcvarsall.bat" amd64 -``` -then from the caffe source folder you need to configure the cmake build -``` -> set CMAKE_GENERATOR=Ninja -> set CMAKE_CONFIGURATION=Release -> mkdir build -> cd build -> cmake -G%CMAKE_GENERATOR% -DBLAS=Open -DCMAKE_BUILD_TYPE=%CMAKE_CONFIGURATION% -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX= -C %CAFFE_DEPENDENCIES%\caffe-builder-config.cmake ..\ -> cmake --build . --config %CMAKE_CONFIGURATION% -> cmake --build . --config %CMAKE_CONFIGURATION% --target install -``` -In the above command `CMAKE_GENERATOR` can be either `Ninja`, `"Visual Studio 12 2013 Win64"` or `"Visual Studio 14 2015 Win64"` and `CMAKE_CONFIGURATION` can be `Release` or `Debug`. Please note however that Visual Studio will not parallelize the build of the CUDA files which results in much longer build times. - -In case one of the steps in the above procedure is not working please refer to the appveyor build scripts in `%CAFFE_ROOT%\scripts\appveyor` to see the most up to date build procedure. - ### Use cuDNN -To use cuDNN you need to define the CUDNN_ROOT cache variable to point to where you unpacked the cuDNN files e.g. `C:/Users/myuser/Projects/machine-learning/cudnn-8.0-windows10-x64-v5.1/cuda`. For example, the build command above would become: - +To use cuDNN you need to define the CUDNN_ROOT cache variable to point to where you unpacked the cuDNN files e.g. `C:/Projects/caffe/cudnn-8.0-windows10-x64-v5.1/cuda`. For example the command in [scripts/build_win.cmd](scripts/build_win.cmd) would become: ``` -> set CMAKE_GENERATOR=Ninja -> set CMAKE_CONFIGURATION=Release -> mkdir build -> cd build -> cmake -G%CMAKE_GENERATOR% -DBLAS=Open -DCMAKE_BUILD_TYPE=%CMAKE_CONFIGURATION% -DBUILD_SHARED_LIBS=OFF -DCMAKE_INSTALL_PREFIX= -DCUDNN_ROOT=C:/Users/myuser/Projects/machine-learning/cudnn-8.0-windows10-x64-v5.1/cuda -C %CAFFE_DEPENDENCIES%\caffe-builder-config.cmake ..\ -> cmake --build . --config %CMAKE_CONFIGURATION% -> cmake --build . --config %CMAKE_CONFIGURATION% --target install +cmake -G"!CMAKE_GENERATOR!" ^ + -DBLAS=Open ^ + -DCMAKE_BUILD_TYPE:STRING=%CMAKE_CONFIG% ^ + -DBUILD_SHARED_LIBS:BOOL=%CMAKE_BUILD_SHARED_LIBS% ^ + -DBUILD_python:BOOL=%BUILD_PYTHON% ^ + -DBUILD_python_layer:BOOL=%BUILD_PYTHON_LAYER% ^ + -DBUILD_matlab:BOOL=%BUILD_MATLAB% ^ + -DCPU_ONLY:BOOL=%CPU_ONLY% ^ + -DCUDNN_ROOT=C:/Projects/caffe/cudnn-8.0-windows10-x64-v5.1/cuda ^ + -C "%cd%\libraries\caffe-builder-config.cmake" ^ + "%~dp0\.." ``` -Make sure to use forward slashes (`/`) in the path. You will need to add the folder containing the cuDNN DLL to your PATH. + +Alternatively, you can open `cmake-gui.exe` and set the variable from there and click `Generate`. ### Building only for CPU @@ -86,30 +87,25 @@ also you will need a protobuf python package that is compatible with pre-built d conda config --add channels willyd conda install --yes protobuf==3.1.0.vc12 ``` -If Python is installed the default is to build the python interface and python layers. If you wish to disable the python layers or the python build use the CMake options `-DBUILD_python_layer=0` and `-DBUILD_python=0` respectively. In order to use the python interface you need to either add the `%CAFFE_ROOT%\python` folder to your python path of copy the `%CAFFE_ROOT%\python\caffe` folder to your `site_packages` folder. Also, you need to edit your `PATH` or copy the required DLLs next to the `caffe.pyd` file. Only Python 2.7 x64 has been tested on Windows. +If Python is installed the default is to build the python interface and python layers. If you wish to disable the python layers or the python build use the CMake options `-DBUILD_python_layer=0` and `-DBUILD_python=0` respectively. In order to use the python interface you need to either add the `C:\Projects\caffe\python` folder to your python path of copy the `C:\Projects\caffe\python\caffe` folder to your `site_packages` folder. Also, you need to edit your `PATH` or copy the required DLLs next to the `caffe.pyd` file. Only Python 2.7 x64 has been tested on Windows. ### Using the MATLAB interface TODO -### Building a shared library -CMake can be used to build a shared library instead of the default static library. To do so follow the above procedure and use `-DBUILD_SHARED_LIBS=ON`. Please note however, that some tests (more specifically the solver related tests) will fail since both the test exectuable and caffe library do not share static objects contained in the protobuf library. +### Using the Ninja generator -### Running the tests or the caffe exectuable - -To run the tests or any caffe exectuable you will have to update your `PATH` to include the directories where the depedencies dlls are located: -``` -:: Prepend to avoid conflicts with other libraries with same name -:: For VS 2013 -> set PATH=%CAFFE_DEPENDENCIES%\bin;%CAFFE_DEPENDENCIES%\lib;%CAFFE_DEPENDENCIES%\x64\vc12\bin;%PATH% -:: For VS 2015 -> set PATH=%CAFFE_DEPENDENCIES%\bin;%CAFFE_DEPENDENCIES%\lib;%CAFFE_DEPENDENCIES%\x64\vc14\bin;%PATH% -``` -or you can use the prependpath.bat included with the prebuilt dependencies. Then the tests can be run from the build folder: -``` -cmake --build . --target runtest --config %CMAKE_CONFIGURATION% +You can choose to use the Ninja generator instead of Visual Studio for faster builds. To do so, change the option `set WITH_NINJA=1` in the `build_win.cmd` script. To install Ninja you can download the executable from github or install it via conda: +```cmd +> conda config --add channels conda-forge +> conda install ninja --yes ``` +When working with ninja you don't have the Visual Studio solutions as ninja is more akin to make. An alternative is to use [Visual Studio Code](https://code.visualstudio.com) with the CMake extensions and C++ extensions. + +### Building a shared library + +CMake can be used to build a shared library instead of the default static library. To do so follow the above procedure and use `-DBUILD_SHARED_LIBS=ON`. Please note however, that some tests (more specifically the solver related tests) will fail since both the test exectuable and caffe library do not share static objects contained in the protobuf library. ### TODOs - Python 3.5: Create protobuf packages for 3.5. Rebuild dependencies especially boost python with 3.5. diff --git a/appveyor.yml b/appveyor.yml index f44227ed713..3fc9a9fdc32 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -2,53 +2,46 @@ version: 1.0.{build} clone_folder: c:\projects\caffe environment: matrix: - - WITH_CMAKE: 1 - MSVC_VERSION: 14 - CMAKE_GENERATOR: Ninja + - MSVC_VERSION: 14 + WITH_NINJA: 0 CMAKE_CONFIG: Release - CMAKE_BUILD_SHARED_LIBS: OFF + CMAKE_BUILD_SHARED_LIBS: 0 - - WITH_CMAKE: 1 - MSVC_VERSION: 14 - CMAKE_GENERATOR: Ninja + - MSVC_VERSION: 14 + WITH_NINJA: 0 CMAKE_CONFIG: Debug - CMAKE_BUILD_SHARED_LIBS: OFF + CMAKE_BUILD_SHARED_LIBS: 0 - - WITH_CMAKE: 1 - MSVC_VERSION: 14 - CMAKE_GENERATOR: Visual Studio 14 2015 Win64 + - MSVC_VERSION: 14 + WITH_NINJA: 1 CMAKE_CONFIG: Release - CMAKE_BUILD_SHARED_LIBS: OFF + CMAKE_BUILD_SHARED_LIBS: 0 - - WITH_CMAKE: 1 - MSVC_VERSION: 14 - CMAKE_GENERATOR: Visual Studio 14 2015 Win64 + - MSVC_VERSION: 14 + WITH_NINJA: 1 CMAKE_CONFIG: Debug - CMAKE_BUILD_SHARED_LIBS: OFF - - WITH_CMAKE: 1 - MSVC_VERSION: 12 - CMAKE_GENERATOR: Ninja + CMAKE_BUILD_SHARED_LIBS: 0 + + - MSVC_VERSION: 12 + WITH_NINJA: 0 CMAKE_CONFIG: Release - CMAKE_BUILD_SHARED_LIBS: OFF + CMAKE_BUILD_SHARED_LIBS: 0 - - WITH_CMAKE: 1 - MSVC_VERSION: 12 - CMAKE_GENERATOR: Ninja + - MSVC_VERSION: 12 + WITH_NINJA: 0 CMAKE_CONFIG: Debug - CMAKE_BUILD_SHARED_LIBS: OFF + CMAKE_BUILD_SHARED_LIBS: 0 - - WITH_CMAKE: 1 - MSVC_VERSION: 12 - CMAKE_GENERATOR: Visual Studio 12 2013 Win64 + - MSVC_VERSION: 12 + WITH_NINJA: 1 CMAKE_CONFIG: Release - CMAKE_BUILD_SHARED_LIBS: OFF + CMAKE_BUILD_SHARED_LIBS: 0 - - WITH_CMAKE: 1 - MSVC_VERSION: 12 - CMAKE_GENERATOR: Visual Studio 12 2013 Win64 + - MSVC_VERSION: 12 + WITH_NINJA: 1 CMAKE_CONFIG: Debug - CMAKE_BUILD_SHARED_LIBS: OFF + CMAKE_BUILD_SHARED_LIBS: 0 build_script: - cmd: >- - call scripts\appveyor\appveyor_build_and_test.cmd \ No newline at end of file + call scripts\build_win.cmd \ No newline at end of file diff --git a/scripts/appveyor/appveyor_build_and_test.cmd b/scripts/appveyor/appveyor_build_and_test.cmd deleted file mode 100644 index 19c7cf41b6f..00000000000 --- a/scripts/appveyor/appveyor_build_and_test.cmd +++ /dev/null @@ -1,7 +0,0 @@ -if "%WITH_CMAKE%" == "1" ( - echo "Building with CMake" - call %~dp0appveyor_cmake_build_and_test.cmd -) else ( - echo "Building with Visual Studio" - call %~dp0appveyor_vs_build_and_test.cmd -) diff --git a/scripts/appveyor/appveyor_cmake_build_and_test.cmd b/scripts/appveyor/appveyor_cmake_build_and_test.cmd deleted file mode 100644 index b5307af6e01..00000000000 --- a/scripts/appveyor/appveyor_cmake_build_and_test.cmd +++ /dev/null @@ -1,76 +0,0 @@ -@echo off - -:: Set python 2.7 with conda as the default python -set PATH=C:\Miniconda-x64;C:\Miniconda-x64\Scripts;C:\Miniconda-x64\Library\bin;%PATH% -:: Check that we have the right python version -python --version -:: Add the required channels -conda config --add channels conda-forge -conda config --add channels willyd -:: Update conda -conda update conda -y -:: Create an environment -:: Todo create protobuf package for vc14 -conda install --yes cmake ninja numpy scipy protobuf==3.1.0.vc12 six scikit-image - -:: Create build directory and configure cmake -mkdir build -pushd build -:: Download dependencies from VS x64 -python ..\scripts\download_prebuilt_dependencies.py --msvc_version v%MSVC_VERSION%0 -:: Add the dependencies to the PATH -:: Prepending is crucial since the hdf5 dll may conflict with python's -call %cd%\libraries\prependpath.bat -:: Setup the environement for VS x64 -@setlocal EnableDelayedExpansion -set batch_file=!VS%MSVC_VERSION%0COMNTOOLS!..\..\VC\vcvarsall.bat -@endlocal & set batch_file=%batch_file% -call "%batch_file%" amd64 -:: Configure using cmake and using the caffe-builder dependencies -cmake -G"%CMAKE_GENERATOR%" ^ - -DBLAS=Open ^ - -DCMAKE_BUILD_TYPE=%CMAKE_CONFIG% ^ - -DBUILD_SHARED_LIBS=%CMAKE_BUILD_SHARED_LIBS% ^ - -C libraries\caffe-builder-config.cmake ^ - ..\ - -:: Build the library and tools -cmake --build . --config %CMAKE_CONFIG% - -if ERRORLEVEL 1 ( - echo Build failed - exit /b 1 -) - -:: Build and exectute the tests -if "%CMAKE_BUILD_SHARED_LIBS%"=="OFF" ( - :: Run the tests only for static lib as the shared lib is causing an issue. - cmake --build . --target runtest --config %CMAKE_CONFIG% - - if ERRORLEVEL 1 ( - echo Tests failed - exit /b 1 - ) - - :: Run python tests only in Release build since - :: the _caffe module is _caffe-d is debug - if "%CMAKE_CONFIG%"=="Release" ( - :: Run the python tests - cmake --build . --target pytest - - if ERRORLEVEL 1 ( - echo Python tests failed - exit /b 1 - ) - ) -) - -:: Lint -cmake --build . --target lint --config %CMAKE_CONFIG% - -if ERRORLEVEL 1 ( - echo Lint failed - exit /b 1 -) - -popd \ No newline at end of file diff --git a/scripts/appveyor/appveyor_vs_build_and_test.cmd b/scripts/appveyor/appveyor_vs_build_and_test.cmd deleted file mode 100644 index 72194a7aa86..00000000000 --- a/scripts/appveyor/appveyor_vs_build_and_test.cmd +++ /dev/null @@ -1,17 +0,0 @@ -@echo off - -cd C:\projects\caffe\windows - -copy CommonSettings.props.example CommonSettings.props - -nuget restore Caffe.sln -PackagesDirectory ..\..\NugetPackages -ConfigFile nuget.config - -set PATH=%PATH:nuget=hello% - -msbuild Caffe.sln /m /v:m /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" /p:Configuration=Debug;CpuOnlyBuild=true;UseCuDNN=false - -msbuild Caffe.sln /m /v:m /logger:"C:\Program Files\AppVeyor\BuildAgent\Appveyor.MSBuildLogger.dll" /p:Configuration=Release;CpuOnlyBuild=true;UseCuDNN=false;WholeProgramOptimization=false - -cd .. - -Build\x64\Release\test_all.exe --gtest_filter=-*TestTimer* \ No newline at end of file diff --git a/scripts/build_win.cmd b/scripts/build_win.cmd new file mode 100644 index 00000000000..4a21aefb2a9 --- /dev/null +++ b/scripts/build_win.cmd @@ -0,0 +1,206 @@ +@echo off +@setlocal EnableDelayedExpansion + +:: Default values +if DEFINED APPVEYOR ( + echo Setting Appveyor defaults + if NOT DEFINED MSVC_VERSION set MSVC_VERSION=14 + if NOT DEFINED WITH_NINJA set WITH_NINJA=1 + if NOT DEFINED CPU_ONLY set CPU_ONLY=1 + if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release + if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0 + if NOT DEFINED BUILD_PYTHON set BUILD_PYTHON=1 + if NOT DEFINED BUILD_PYTHON_LAYER set BUILD_PYTHON_LAYER=1 + if NOT DEFINED BUILD_MATLAB set BUILD_MATLAB=0 + if NOT DEFINED PYTHON_EXE set PYTHON_EXE=python + if NOT DEFINED RUN_TESTS set RUN_TESTS=1 + if NOT DEFINED RUN_LINT set RUN_LINT=1 + if NOT DEFINED RUN_INSTALL set RUN_INSTALL=1 + + :: Set python 2.7 with conda as the default python + set PATH=C:\Miniconda-x64;C:\Miniconda-x64\Scripts;C:\Miniconda-x64\Library\bin;!PATH! + :: Check that we have the right python version + !PYTHON_EXE! --version + :: Add the required channels + conda config --add channels conda-forge + conda config --add channels willyd + :: Update conda + conda update conda -y + :: Create an environment + :: Todo create protobuf package for vc14 + conda install --yes cmake ninja numpy scipy protobuf==3.1.0.vc12 six scikit-image + + :: Disable the tests in debug config + if "%CMAKE_CONFIG%" == "Debug" ( + echo Disabling tests on appveyor with config == %CMAKE_CONFIG% + set RUN_TESTS=0 + ) + +) else ( + :: Change the settings here to match your setup + :: Change MSVC_VERSION to 12 to use VS 2013 + if NOT DEFINED MSVC_VERSION set MSVC_VERSION=14 + :: Change to 1 to use Ninja generator (builds much faster) + if NOT DEFINED WITH_NINJA set WITH_NINJA=0 + :: Change to 1 to build caffe without CUDA support + if NOT DEFINED CPU_ONLY set CPU_ONLY=0 + :: Change to Debug to build Debug. This is only relevant for the Ninja generator the Visual Studio generator will generate both Debug and Release configs + if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release + :: Change to 1 to build a caffe.dll + if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0 + :: Change these options for your needs. + if NOT DEFINED BUILD_PYTHON set BUILD_PYTHON=1 + if NOT DEFINED BUILD_PYTHON_LAYER set BUILD_PYTHON_LAYER=1 + if NOT DEFINED BUILD_MATLAB set BUILD_MATLAB=0 + :: If python is on your path leave this alone + if NOT DEFINED PYTHON_EXE set PYTHON_EXE=python + :: Run the tests + if NOT DEFINED RUN_TESTS set RUN_TESTS=0 + :: Run lint + if NOT DEFINED RUN_LINT set RUN_LINT=0 + :: Build the install target + if NOT DEFINED RUN_INSTALL set RUN_INSTALL=0 +) + +:: Set the appropriate CMake generator +:: Use the exclamation mark ! below to delay the +:: expansion of CMAKE_GENERATOR +if %WITH_NINJA% EQU 0 ( + if "%MSVC_VERSION%"=="14" ( + set CMAKE_GENERATOR=Visual Studio 14 2015 Win64 + ) + if "%MSVC_VERSION%"=="12" ( + set CMAKE_GENERATOR=Visual Studio 12 2013 Win64 + ) + if "!CMAKE_GENERATOR!"=="" ( + echo ERROR: Unsupported MSVC version + exit /B 1 + ) +) else ( + set CMAKE_GENERATOR=Ninja +) + +echo INFO: ============================================================ +echo INFO: Summary: +echo INFO: ============================================================ +echo INFO: MSVC_VERSION = !MSVC_VERSION! +echo INFO: WITH_NINJA = !WITH_NINJA! +echo INFO: CMAKE_GENERATOR = "!CMAKE_GENERATOR!" +echo INFO: CPU_ONLY = !CPU_ONLY! +echo INFO: CMAKE_CONFIG = !CMAKE_CONFIG! +echo INFO: CMAKE_BUILD_SHARED_LIBS = !CMAKE_BUILD_SHARED_LIBS! +echo INFO: BUILD_PYTHON = !BUILD_PYTHON! +echo INFO: BUILD_PYTHON_LAYER = !BUILD_PYTHON_LAYER! +echo INFO: BUILD_MATLAB = !BUILD_MATLAB! +echo INFO: PYTHON_EXE = "!PYTHON_EXE!" +echo INFO: RUN_TESTS = !RUN_TESTS! +echo INFO: RUN_LINT = !RUN_LINT! +echo INFO: RUN_INSTALL = !RUN_INSTALL! +echo INFO: ============================================================ + +:: Build and exectute the tests +:: Do not run the tests with shared library +if !RUN_TESTS! EQU 1 ( + if %CMAKE_BUILD_SHARED_LIBS% EQU 1 ( + echo WARNING: Disabling tests with shared library build + set RUN_TESTS=0 + ) +) + +:: Create build directory and configure cmake +if EXIST build ( + echo ERROR: build directory already exists in %cd%\build please remove it and start over. + exit /b 1 +) + +mkdir build +pushd build + +:: Download dependencies from VS x64 +echo INFO: Downloading dependencies +"%PYTHON_EXE%" "%~dp0\download_prebuilt_dependencies.py" --msvc_version v%MSVC_VERSION%0 + +if ERRORLEVEL 1 ( + echo ERROR: Downloading dependencies failed + exit /b 1 +) + + +:: Add the dependencies to the PATH +if EXIST "%cd%\libraries\prependpath.bat" ( + call "%cd%\libraries\prependpath.bat" +) + +:: Setup the environement for VS x64 +set batch_file=!VS%MSVC_VERSION%0COMNTOOLS!..\..\VC\vcvarsall.bat +call "%batch_file%" amd64 + +:: Configure using cmake and using the caffe-builder dependencies +:: Add -DCUDNN_ROOT=C:/Projects/caffe/cudnn-8.0-windows10-x64-v5.1/cuda ^ +:: below to use cuDNN +cmake -G"!CMAKE_GENERATOR!" ^ + -DBLAS=Open ^ + -DCMAKE_BUILD_TYPE:STRING=%CMAKE_CONFIG% ^ + -DBUILD_SHARED_LIBS:BOOL=%CMAKE_BUILD_SHARED_LIBS% ^ + -DBUILD_python:BOOL=%BUILD_PYTHON% ^ + -DBUILD_python_layer:BOOL=%BUILD_PYTHON_LAYER% ^ + -DBUILD_matlab:BOOL=%BUILD_MATLAB% ^ + -DCPU_ONLY:BOOL=%CPU_ONLY% ^ + -C "%cd%\libraries\caffe-builder-config.cmake" ^ + "%~dp0\.." + +if ERRORLEVEL 1 ( + echo Configure failed + exit /b 1 +) + +:: Lint +if %RUN_LINT% EQU 1 ( + cmake --build . --target lint --config %CMAKE_CONFIG% +) + +if ERRORLEVEL 1 ( + echo Lint failed + exit /b 1 +) + +:: Build the library and tools +cmake --build . --config %CMAKE_CONFIG% + +if ERRORLEVEL 1 ( + echo Build failed + exit /b 1 +) + +:: Build and exectute the tests +if !RUN_TESTS! EQU 1 ( + cmake --build . --target runtest --config %CMAKE_CONFIG% + + if ERRORLEVEL 1 ( + echo Tests failed + exit /b 1 + ) + + if %BUILD_PYTHON% EQU 1 ( + if %BUILD_PYTHON_LAYER% EQU 1 ( + :: Run python tests only in Release build since + :: the _caffe module is _caffe-d is debug + if "%CMAKE_CONFIG%"=="Release" ( + :: Run the python tests + cmake --build . --target pytest + + if ERRORLEVEL 1 ( + echo Python tests failed + exit /b 1 + ) + ) + ) + ) +) + +if %RUN_INSTALL% EQU 1 ( + cmake --build . --target install --config %CMAKE_CONFIG% +) + +popd +@endlocal \ No newline at end of file diff --git a/scripts/download_prebuilt_dependencies.py b/scripts/download_prebuilt_dependencies.py index be5397ba71e..f3d31147ca0 100644 --- a/scripts/download_prebuilt_dependencies.py +++ b/scripts/download_prebuilt_dependencies.py @@ -11,12 +11,14 @@ from six.moves import urllib from download_model_binary import reporthook -WIN_DEPENDENCIES_URLS = dict( - v120=("https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v120_x64_py27_1.0.1.tar.bz2", - "3f45fe3f27b27a7809f9de1bd85e56888b01dbe2"), - v140=("https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v140_x64_py27_1.0.1.tar.bz2", - "427faf33745cf8cd70c7d043c85db7dda7243122"), -) +WIN_DEPENDENCIES_URLS = { + ('v120', '2.7'):("https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v120_x64_py27_1.0.1.tar.bz2", + "3f45fe3f27b27a7809f9de1bd85e56888b01dbe2"), + ('v140', '2.7'):("https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v140_x64_py27_1.0.1.tar.bz2", + "427faf33745cf8cd70c7d043c85db7dda7243122"), + ('v140', '3.5'):("https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v140_x64_py35_1.0.1.tar.bz2", + "1f55dac54aeab7ae3a1cda145ca272dea606bdf9"), +} # function for checking SHA1. def model_checks_out(filename, sha1): @@ -31,15 +33,18 @@ def model_checks_out(filename, sha1): args = parser.parse_args() # get the appropriate url + pyver = '{:d}.{:d}'.format(sys.version_info.major, sys.version_info.minor) try: - url, sha1 = WIN_DEPENDENCIES_URLS[args.msvc_version] + url, sha1 = WIN_DEPENDENCIES_URLS[(args.msvc_version, pyver)] except KeyError: - print('ERROR: Could not find url for MSVC version = {}.'.format(args.msvc_version)) + print('ERROR: Could not find url for MSVC version = {} and Python version = {}.\n{}' + .format(args.msvc_version, pyver, + 'Available combinations are: {}'.format(list(WIN_DEPENDENCIES_URLS.keys())))) sys.exit(1) dep_filename = os.path.split(url)[1] # Download binaries - print("Downloading dependencies. Please wait...") + print("Downloading dependencies ({}). Please wait...".format(dep_filename)) urllib.request.urlretrieve(url, dep_filename, reporthook) if not model_checks_out(dep_filename, sha1): print('ERROR: dependencies did not download correctly! Run this again.') From a61b7b98fbe78f01ceca403fcb696c16277e08cb Mon Sep 17 00:00:00 2001 From: twmht Date: Tue, 29 Nov 2016 20:58:00 +0800 Subject: [PATCH 25/51] fix typo when including cmake files of opencv --- cmake/Templates/CaffeConfig.cmake.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/Templates/CaffeConfig.cmake.in b/cmake/Templates/CaffeConfig.cmake.in index 243c9a6a70b..dd53b9f704e 100644 --- a/cmake/Templates/CaffeConfig.cmake.in +++ b/cmake/Templates/CaffeConfig.cmake.in @@ -30,7 +30,7 @@ if(@USE_OPENCV@) if(MSVC) # The path to OpenCVModules.cmake is mangled according to # compiler and arch on Windows - include(${Caffe_OpenCV_CONFIG_PATH}/OpenConfig.cmake) + include(${Caffe_OpenCV_CONFIG_PATH}/OpenCVConfig.cmake) else() include(${Caffe_OpenCV_CONFIG_PATH}/OpenCVModules.cmake) endif() From 4ef2c3e07c460a54cebd8ea9672732bb7e498c15 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Wed, 30 Nov 2016 20:00:07 -0500 Subject: [PATCH 26/51] Added artifacts to appveyor builds --- appveyor.yml | 8 +++++++- scripts/build_win.cmd | 15 ++++++++++----- 2 files changed, 17 insertions(+), 6 deletions(-) diff --git a/appveyor.yml b/appveyor.yml index 3fc9a9fdc32..d7623be5e43 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -44,4 +44,10 @@ environment: build_script: - cmd: >- - call scripts\build_win.cmd \ No newline at end of file + call scripts\build_win.cmd + +artifacts: + - path: build\install + name: caffe + - path: build\libraries + name: dependencies \ No newline at end of file diff --git a/scripts/build_win.cmd b/scripts/build_win.cmd index 4a21aefb2a9..98a81bdd021 100644 --- a/scripts/build_win.cmd +++ b/scripts/build_win.cmd @@ -30,6 +30,11 @@ if DEFINED APPVEYOR ( :: Todo create protobuf package for vc14 conda install --yes cmake ninja numpy scipy protobuf==3.1.0.vc12 six scikit-image + if ERRORLEVEL 1 ( + echo ERROR: Conda update or install failed + exit /b 1 + ) + :: Disable the tests in debug config if "%CMAKE_CONFIG%" == "Debug" ( echo Disabling tests on appveyor with config == %CMAKE_CONFIG% @@ -150,7 +155,7 @@ cmake -G"!CMAKE_GENERATOR!" ^ "%~dp0\.." if ERRORLEVEL 1 ( - echo Configure failed + echo ERROR: Configure failed exit /b 1 ) @@ -160,7 +165,7 @@ if %RUN_LINT% EQU 1 ( ) if ERRORLEVEL 1 ( - echo Lint failed + echo ERROR: Lint failed exit /b 1 ) @@ -168,7 +173,7 @@ if ERRORLEVEL 1 ( cmake --build . --config %CMAKE_CONFIG% if ERRORLEVEL 1 ( - echo Build failed + echo ERROR: Build failed exit /b 1 ) @@ -177,7 +182,7 @@ if !RUN_TESTS! EQU 1 ( cmake --build . --target runtest --config %CMAKE_CONFIG% if ERRORLEVEL 1 ( - echo Tests failed + echo ERROR: Tests failed exit /b 1 ) @@ -190,7 +195,7 @@ if !RUN_TESTS! EQU 1 ( cmake --build . --target pytest if ERRORLEVEL 1 ( - echo Python tests failed + echo ERROR: Python tests failed exit /b 1 ) ) From 3649015cd1b78cd435a7f624e9a6a68454da8880 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Sun, 8 Jan 2017 21:27:15 -0500 Subject: [PATCH 27/51] Added missing pyyaml in appveyor build script --- scripts/build_win.cmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/build_win.cmd b/scripts/build_win.cmd index 98a81bdd021..d408e7048d1 100644 --- a/scripts/build_win.cmd +++ b/scripts/build_win.cmd @@ -28,7 +28,7 @@ if DEFINED APPVEYOR ( conda update conda -y :: Create an environment :: Todo create protobuf package for vc14 - conda install --yes cmake ninja numpy scipy protobuf==3.1.0.vc12 six scikit-image + conda install --yes cmake ninja numpy scipy protobuf==3.1.0.vc12 six scikit-image pyyaml if ERRORLEVEL 1 ( echo ERROR: Conda update or install failed From 38862a5fc25c987662b5ecba560c10bb5dc80067 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Sun, 8 Jan 2017 21:56:37 -0500 Subject: [PATCH 28/51] Fixed appveyor build status badge and added prebuilt binaries download link --- README.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 58b6e2aed16..a025005cece 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,16 @@ This branch of Caffe ports the framework to Windows. [![Travis Build Status](https://api.travis-ci.org/BVLC/caffe.svg?branch=windows)](https://travis-ci.org/BVLC/caffe) Travis (Linux build) -[![Windows Build status](https://ci.appveyor.com/api/projects/status/6xpwyq0y9ffdj9pb/branch/windows?svg=true)](https://ci.appveyor.com/project/willyd/caffe-4pvka/branch/windows) AppVeyor (Windows build) +[![Build status](https://ci.appveyor.com/api/projects/status/ew7cl2k1qfsnyql4/branch/windows?svg=true)](https://ci.appveyor.com/project/BVLC/caffe/branch/windows) AppVeyor (Windows build) + +## Prebuilt binaries + +Prebuilt binaries can be downloaded from the latest CI build on appveyor for the following configurations: + +- Visual Studio 2015, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DDebug%2C+CMAKE_BUILD_SHARED_LIBS%3D0) and [Caffe Dependencies](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/dependencies.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0) + +- Visual Studio 2013, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D12%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D12%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DDebug%2C+CMAKE_BUILD_SHARED_LIBS%3D0) and [Caffe Dependencies](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/dependencies.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D12%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0) + ## Windows Setup From 58563abfae2d7bd055fa44185643f79c98c5fab6 Mon Sep 17 00:00:00 2001 From: Luis Unzueta Date: Sun, 30 Oct 2016 17:57:26 +0100 Subject: [PATCH 29/51] Update CMake files to build Matcaffe on Windows --- CMakeLists.txt | 2 +- README.md | 2 +- cmake/Dependencies.cmake | 14 ++++++++---- matlab/CMakeLists.txt | 59 ++++++++++++++++++++++++++++++++---------------- 4 files changed, 51 insertions(+), 26 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index dd8cfdb8c89..6b2a17c095d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -45,7 +45,7 @@ else() endif() caffe_option(BUILD_python "Build Python wrapper" ON) set(python_version "2" CACHE STRING "Specify which Python version to use") -caffe_option(BUILD_matlab "Build Matlab wrapper" OFF IF UNIX OR APPLE) +caffe_option(BUILD_matlab "Build Matlab wrapper" OFF) caffe_option(BUILD_docs "Build documentation" ON IF UNIX OR APPLE) caffe_option(BUILD_python_layer "Build the Caffe Python layer" ON) caffe_option(USE_OPENCV "Build with OpenCV support" ON) diff --git a/README.md b/README.md index a025005cece..a6dde4b45f9 100644 --- a/README.md +++ b/README.md @@ -100,7 +100,7 @@ If Python is installed the default is to build the python interface and python l ### Using the MATLAB interface -TODO +Follow the above procedure and use `-DBUILD_matlab=ON`. Then, you need to add the path to the generated `.mexw64` file to your `PATH` and the folder caffe/matlab to your Matlab search PATH to use matcaffe. ### Using the Ninja generator diff --git a/cmake/Dependencies.cmake b/cmake/Dependencies.cmake index 16d701d0f64..50dfd7b0d15 100644 --- a/cmake/Dependencies.cmake +++ b/cmake/Dependencies.cmake @@ -178,11 +178,17 @@ endif() # ---[ Matlab if(BUILD_matlab) - find_package(MatlabMex) - if(MATLABMEX_FOUND) - set(HAVE_MATLAB TRUE) + if(MSVC) + find_package(Matlab COMPONENTS MAIN_PROGRAM MX_LIBRARY) + if(MATLAB_FOUND) + set(HAVE_MATLAB TRUE) + endif() + else() + find_package(MatlabMex) + if(MATLABMEX_FOUND) + set(HAVE_MATLAB TRUE) + endif() endif() - # sudo apt-get install liboctave-dev find_program(Octave_compiler NAMES mkoctfile DOC "Octave C++ compiler") diff --git a/matlab/CMakeLists.txt b/matlab/CMakeLists.txt index 987730d9b55..c81c2af8845 100644 --- a/matlab/CMakeLists.txt +++ b/matlab/CMakeLists.txt @@ -1,5 +1,5 @@ # Builds Matlab (or Octave) interface. In case of Matlab caffe must be -# compield as shared library. Octave can link static or shared caffe library +# compiled as shared library. Octave can link static or shared caffe library # To install octave run: sudo apt-get install liboctave-dev if(NOT BUILD_matlab) @@ -16,26 +16,34 @@ else() return() endif() -if(NOT BUILD_SHARED_LIBS AND build_using MATCHES Matlab) - message(FATAL_ERROR "Matlab MEX interface (with default mex options file) can only be built if caffe is compiled as shared library. Please enable 'BUILD_SHARED_LIBS' in CMake. Aternativelly you can switch to Octave compiler.") +if(NOT MSVC AND NOT BUILD_SHARED_LIBS AND build_using MATCHES Matlab) + message(FATAL_ERROR "Matlab MEX interface (with default mex options file) can only be built if caffe is compiled as shared library on UNIX systems. Please enable 'BUILD_SHARED_LIBS' in CMake. Alternatively you can switch to Octave compiler.") endif() +if(NOT MSVC) # helper function to set proper mex file extension -function(caffe_fetch_and_set_proper_mexext mexfile_variable) - execute_process(COMMAND ${Matlab_mexext} OUTPUT_STRIP_TRAILING_WHITESPACE RESULT_VARIABLE res OUTPUT_VARIABLE ext) - if(res MATCHES 0) - get_filename_component(folder ${${mexfile_variable}} PATH) - get_filename_component(name_we ${${mexfile_variable}} NAME_WE) - set(${mexfile_variable} ${folder}/${name_we}.${ext} PARENT_SCOPE) - endif() -endfunction() + function(caffe_fetch_and_set_proper_mexext mexfile_variable) + execute_process(COMMAND ${Matlab_mexext} OUTPUT_STRIP_TRAILING_WHITESPACE RESULT_VARIABLE res OUTPUT_VARIABLE ext) + if(res MATCHES 0) + get_filename_component(folder ${${mexfile_variable}} PATH) + get_filename_component(name_we ${${mexfile_variable}} NAME_WE) + set(${mexfile_variable} ${folder}/${name_we}.${ext} PARENT_SCOPE) + endif() + endfunction() +endif() # global settings file(GLOB Matlab_srcs +caffe/private/caffe_.cpp) -set(Matlab_caffe_mex ${PROJECT_SOURCE_DIR}/matlab/+caffe/private/caffe_.mex) +if(MSVC) + set(Matlab_caffe_mex ${PROJECT_SOURCE_DIR}/matlab/+caffe/private/caffe_.mexw64) +else() + set(Matlab_caffe_mex ${PROJECT_SOURCE_DIR}/matlab/+caffe/private/caffe_.mex) +endif() caffe_get_current_cflags(cflags) -caffe_parse_linker_libs(Caffe_LINKER_LIBS folders libflags macos_frameworks) +if(NOT MSVC) + caffe_parse_linker_libs(Caffe_LINKER_LIBS folders libflags macos_frameworks) +endif() set(folders $ ${folders}) # prepare linker flag lists @@ -43,14 +51,25 @@ string(REPLACE ";" ";-L" link_folders "-L${folders}") string(REPLACE ";" ":" rpath_folders "${folders}") if(build_using MATCHES "Matlab") - set(libflags -lcaffe${Caffe_POSTFIX} ${libflags}) # Matlab R2014a complans for -Wl,--whole-archive - - caffe_fetch_and_set_proper_mexext(Matlab_caffe_mex) - add_custom_command(OUTPUT ${Matlab_caffe_mex} COMMAND ${Matlab_mex} - ARGS -output ${Matlab_caffe_mex} ${Matlab_srcs} ${cflags} ${link_folders} ${libflags} - DEPENDS caffe COMMENT "Building Matlab interface: ${Matlab_caffe_mex}" VERBATIM) - add_custom_target(matlab ALL DEPENDS ${Matlab_caffe_mex} SOURCES ${Matlab_srcs}) + if(MSVC) + matlab_add_mex(NAME matlab + SRC ${Matlab_srcs} # maybe you need to add some other sources + OUTPUT_NAME caffe_ # change the output name to _caffe.mexw64 + LINK_TO caffe # cmake will take care of forwarding the correct transitive library dependencies to your mex file + ) + # output the target in the source tree as in the original version. + set_target_properties(matlab PROPERTIES + RUNTIME_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}/matlab/+caffe/private + ) + else() + set(libflags -lcaffe${Caffe_POSTFIX} ${libflags}) # Matlab R2014a complans for -Wl,--whole-archive + caffe_fetch_and_set_proper_mexext(Matlab_caffe_mex) + add_custom_command(OUTPUT ${Matlab_caffe_mex} COMMAND ${Matlab_mex} + ARGS -output ${Matlab_caffe_mex} ${Matlab_srcs} ${cflags} ${link_folders} ${libflags} + DEPENDS caffe COMMENT "Building Matlab interface: ${Matlab_caffe_mex}" VERBATIM) + add_custom_target(matlab ALL DEPENDS ${Matlab_caffe_mex} SOURCES ${Matlab_srcs}) + endif() elseif(build_using MATCHES "Octave") if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "Clang") From 82c087ca05795d23c939e55b1b12f9046140ed5c Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Tue, 13 Dec 2016 09:17:52 -0500 Subject: [PATCH 30/51] Added support for python 3.5 --- CMakeLists.txt | 10 ++ README.md | 55 ++++---- appveyor.yml | 18 ++- cmake/TargetResolvePrerequesites.cmake | 163 ++++++++++++++++++++++++ cmake/WindowsDownloadPrebuiltDependencies.cmake | 75 +++++++++++ python/CMakeLists.txt | 11 +- scripts/build_win.cmd | 74 +++++------ src/caffe/test/CMakeLists.txt | 4 + tools/CMakeLists.txt | 8 ++ 9 files changed, 345 insertions(+), 73 deletions(-) create mode 100644 cmake/TargetResolvePrerequesites.cmake create mode 100644 cmake/WindowsDownloadPrebuiltDependencies.cmake diff --git a/CMakeLists.txt b/CMakeLists.txt index 6b2a17c095d..ed19f0311ae 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -33,6 +33,7 @@ include(cmake/Misc.cmake) include(cmake/Summary.cmake) include(cmake/ConfigGen.cmake) include(cmake/WindowsCreateLinkHeader.cmake) +include(cmake/TargetResolvePrerequesites.cmake) # ---[ Options caffe_option(CPU_ONLY "Build Caffe without CUDA support" OFF) # TODO: rename to USE_CUDA @@ -53,8 +54,14 @@ caffe_option(USE_LEVELDB "Build with levelDB" ON) caffe_option(USE_LMDB "Build with lmdb" ON) caffe_option(ALLOW_LMDB_NOLOCK "Allow MDB_NOLOCK when reading LMDB files (only if necessary)" OFF) caffe_option(protobuf_MODULE_COMPATIBLE "Make the protobuf-config.cmake compatible with the module mode" ON IF MSVC) +caffe_option(COPY_PREREQUISITES "Copy the prerequisites next to each executable or shared library directory" ON IF MSVC) +caffe_option(INSTALL_PREREQUISITES "Install the prerequisites next to each executable or shared library directory" ON IF MSVC) if(MSVC AND BUILD_SHARED_LIBS) + if(CMAKE_GENERATOR MATCHES "Visual Studio") + # see issue https://gitlab.kitware.com/cmake/cmake/issues/16552#note_215236 + message(FATAL_ERROR "The Visual Studio generator cannot build a shared library. Use the Ninja generator instead.") + endif() # Some tests (solver tests) fail when caffe is built as a shared library. The problem comes # from protobuf that has a global static empty_string_ variable. Since caffe and test.testbin # link to a static protobuf library both end up with their own instance of the empty_string_ @@ -63,6 +70,9 @@ if(MSVC AND BUILD_SHARED_LIBS) message(WARNING "Some tests (solvers) will fail when building as a shared library with MSVC") endif() +# ---[ Prebuild dependencies on windows +include(cmake/WindowsDownloadPrebuiltDependencies.cmake) + # ---[ Dependencies include(cmake/Dependencies.cmake) diff --git a/README.md b/README.md index a6dde4b45f9..a1dfc992624 100644 --- a/README.md +++ b/README.md @@ -12,9 +12,11 @@ This branch of Caffe ports the framework to Windows. Prebuilt binaries can be downloaded from the latest CI build on appveyor for the following configurations: -- Visual Studio 2015, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DDebug%2C+CMAKE_BUILD_SHARED_LIBS%3D0) and [Caffe Dependencies](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/dependencies.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0) +- Visual Studio 2015, CPU only, Python 3.5: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D3), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DDebug%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D3) -- Visual Studio 2013, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D12%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D12%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DDebug%2C+CMAKE_BUILD_SHARED_LIBS%3D0) and [Caffe Dependencies](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/dependencies.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D12%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0) +- Visual Studio 2015, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D2), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DDebug%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D2) + +- Visual Studio 2013, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D12%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D2), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D12%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DDebug%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D2) ## Windows Setup @@ -23,9 +25,13 @@ Prebuilt binaries can be downloaded from the latest CI build on appveyor for the - Visual Studio 2013 or 2015 - [CMake](https://cmake.org/) 3.4 or higher (Visual Studio and [Ninja](https://ninja-build.org/) generators are supported) - - Python 2.7 Anaconda x64 (or Miniconda). - - CUDA 7.5 or 8.0 (optional) (use CUDA 8 if using Visual Studio 2015) - - cuDNN v5 (optional) + +### Optional Dependencies + + - Python for the pycaffe interface. Anaconda Python 2.7 or 3.5 x64 (or Miniconda) + - Matlab for the matcaffe interface. + - CUDA 7.5 or 8.0 (use CUDA 8 if using Visual Studio 2015) + - cuDNN v5 We assume that `cmake.exe` and `python.exe` are on your `PATH`. @@ -39,32 +45,17 @@ C:\Projects\caffe> git checkout windows :: Edit any of the options inside build_win.cmd to suit your needs C:\Projects\caffe> scripts\build_win.cmd ``` -The `build_win.cmd` script should be executed once to download the dependencies, create the Visual Studio project files (or the ninja build files) and build the Release configuration. After that you should add the required folders to your `PATH` by executing the following command: -```cmd -C:\Projects\caffe> call build\libraries\prependpath.bat -``` -Once this is done you can use the `pycaffe` interface or run `caffe.exe` from the command line. If you want to debug the `caffe.exe` exectuable, open Visual Studio from a `cmd.exe` prompt that has the required directories in its `PATH` variable and open the `C:\Projects\caffe\build\Caffe.sln` and proceed as normal. Alternatively, you can copy the required DLLs next to the `caffe.exe` ( or `caffe-d.exe` in Debug). - -Should you encounter any error please post the output of the above commands by redirecting the output to a file and open a topic on the [caffe-users list](https://groups.google.com/forum/#!forum/caffe-users) mailing list. +The `build_win.cmd` script will download the dependencies, create the Visual Studio project files (or the ninja build files) and build the Release configuration. By default all the required DLLs will be copied (or hard linked when possible) next to the consuming binaries. If you wish to disable this option, you can by changing the command line option `-DCOPY_PREREQUISITES=0`. The prebuilt libraries also provide a `prependpath.bat` batch script that can temporarily modify your `PATH` envrionment variable to make the required DLLs available. Below is a more complete description of some of the steps involved in building caffe. ### Install the caffe dependencies -The easiest and recommended way of installing the required dependencies is by downloading the pre-built libraries using the [scripts\download_prebuilt_dependencies.py](scripts\download_prebuilt_dependencies.py) file. Depending on your compiler one of the following commands should download and extract the prebuilt dependencies to your current working directory: - -```cmd -:: Install Visual Studio 2013 dependencies -> python scripts\download_prebuilt_dependencies.py --msvc_version=v120 -:: Or install Visual Studio 2015 dependencies -> python scripts\download_prebuilt_dependencies.py --msvc_version=v140 -``` - -This will create a folder called `libraries` containing all the required dependencies. Alternatively you can build them yourself by following the instructions in the [caffe-builder](https://github.com/willyd/caffe-builder) [README](https://github.com/willyd/caffe-builder/blob/master/README.md). For the remaining of these instructions we will assume that the libraries folder is in a folder defined by the `%CAFFE_DEPENDENCIES%` environment variable. +By default CMake will download and extract prebuilt dependencies for your compiler and python version. It will create a folder called `libraries` containing all the required dependencies inside your build folder. Alternatively you can build them yourself by following the instructions in the [caffe-builder](https://github.com/willyd/caffe-builder) [README](https://github.com/willyd/caffe-builder/blob/master/README.md). ### Use cuDNN -To use cuDNN you need to define the CUDNN_ROOT cache variable to point to where you unpacked the cuDNN files e.g. `C:/Projects/caffe/cudnn-8.0-windows10-x64-v5.1/cuda`. For example the command in [scripts/build_win.cmd](scripts/build_win.cmd) would become: +To use cuDNN the easiest way is to copy the content of the `cuda` folder into your CUDA toolkit installation directory. For example if you installed CUDA 8.0 and downloaded cudnn-8.0-windows10-x64-v5.1.zip you should copy the content of the `cuda` directory to `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v8.0`. Alternatively, you can define the CUDNN_ROOT cache variable to point to where you unpacked the cuDNN files e.g. `C:/Projects/caffe/cudnn-8.0-windows10-x64-v5.1/cuda`. For example the command in [scripts/build_win.cmd](scripts/build_win.cmd) would become: ``` cmake -G"!CMAKE_GENERATOR!" ^ -DBLAS=Open ^ @@ -93,15 +84,17 @@ conda install --yes numpy scipy matplotlib scikit-image pip six ``` also you will need a protobuf python package that is compatible with pre-built dependencies. This package can be installed this way: ``` -conda config --add channels willyd -conda install --yes protobuf==3.1.0.vc12 +conda install --yes --channel willyd protobuf==3.1.0 ``` -If Python is installed the default is to build the python interface and python layers. If you wish to disable the python layers or the python build use the CMake options `-DBUILD_python_layer=0` and `-DBUILD_python=0` respectively. In order to use the python interface you need to either add the `C:\Projects\caffe\python` folder to your python path of copy the `C:\Projects\caffe\python\caffe` folder to your `site_packages` folder. Also, you need to edit your `PATH` or copy the required DLLs next to the `caffe.pyd` file. Only Python 2.7 x64 has been tested on Windows. +If Python is installed the default is to build the python interface and python layers. If you wish to disable the python layers or the python build use the CMake options `-DBUILD_python_layer=0` and `-DBUILD_python=0` respectively. In order to use the python interface you need to either add the `C:\Projects\caffe\python` folder to your python path of copy the `C:\Projects\caffe\python\caffe` folder to your `site_packages` folder. ### Using the MATLAB interface -Follow the above procedure and use `-DBUILD_matlab=ON`. Then, you need to add the path to the generated `.mexw64` file to your `PATH` and the folder caffe/matlab to your Matlab search PATH to use matcaffe. - +Follow the above procedure and use `-DBUILD_matlab=ON`. Change your current directory in MATLAB to `C:\Projects\caffe\matlab` and run the following command to run the tests: +``` +>> caffe.run_tests() +``` +If all tests pass you can test if the classification_demo works as well. First, from `C:\Projects\caffe` run `python scripts\download_model_binary.py models\bvlc_reference_caffenet` to download the pre-trained caffemodel from the model zoo. Then change your MATLAB directory to `C:\Projects\caffe\matlab\demo` and run `classification_demo`. ### Using the Ninja generator @@ -116,8 +109,9 @@ When working with ninja you don't have the Visual Studio solutions as ninja is m CMake can be used to build a shared library instead of the default static library. To do so follow the above procedure and use `-DBUILD_SHARED_LIBS=ON`. Please note however, that some tests (more specifically the solver related tests) will fail since both the test exectuable and caffe library do not share static objects contained in the protobuf library. -### TODOs -- Python 3.5: Create protobuf packages for 3.5. Rebuild dependencies especially boost python with 3.5. +### Troubleshooting + +Should you encounter any error please post the output of the above commands by redirecting the output to a file and open a topic on the [caffe-users list](https://groups.google.com/forum/#!forum/caffe-users) mailing list. ## Previous Visual Studio based build @@ -127,6 +121,7 @@ The previous windows build based on Visual Studio project files is now deprecate - The `GPUTimer` related test cases always fail on Windows. This seems to be a difference between UNIX and Windows. - Shared library (DLL) build will have failing tests. +- Shared library build only works with the Ninja generator ## Further Details diff --git a/appveyor.yml b/appveyor.yml index d7623be5e43..5e1de384aa7 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -6,41 +6,55 @@ environment: WITH_NINJA: 0 CMAKE_CONFIG: Release CMAKE_BUILD_SHARED_LIBS: 0 + PYTHON_VERSION: 3 + + - MSVC_VERSION: 14 + WITH_NINJA: 0 + CMAKE_CONFIG: Release + CMAKE_BUILD_SHARED_LIBS: 0 + PYTHON_VERSION: 2 - MSVC_VERSION: 14 WITH_NINJA: 0 CMAKE_CONFIG: Debug CMAKE_BUILD_SHARED_LIBS: 0 + PYTHON_VERSION: 2 - MSVC_VERSION: 14 WITH_NINJA: 1 CMAKE_CONFIG: Release CMAKE_BUILD_SHARED_LIBS: 0 + PYTHON_VERSION: 2 - MSVC_VERSION: 14 WITH_NINJA: 1 CMAKE_CONFIG: Debug CMAKE_BUILD_SHARED_LIBS: 0 + PYTHON_VERSION: 2 - MSVC_VERSION: 12 WITH_NINJA: 0 CMAKE_CONFIG: Release CMAKE_BUILD_SHARED_LIBS: 0 + PYTHON_VERSION: 2 - MSVC_VERSION: 12 WITH_NINJA: 0 CMAKE_CONFIG: Debug CMAKE_BUILD_SHARED_LIBS: 0 + PYTHON_VERSION: 2 - MSVC_VERSION: 12 WITH_NINJA: 1 CMAKE_CONFIG: Release CMAKE_BUILD_SHARED_LIBS: 0 + PYTHON_VERSION: 2 - MSVC_VERSION: 12 WITH_NINJA: 1 CMAKE_CONFIG: Debug CMAKE_BUILD_SHARED_LIBS: 0 + PYTHON_VERSION: 2 build_script: - cmd: >- @@ -48,6 +62,4 @@ build_script: artifacts: - path: build\install - name: caffe - - path: build\libraries - name: dependencies \ No newline at end of file + name: caffe \ No newline at end of file diff --git a/cmake/TargetResolvePrerequesites.cmake b/cmake/TargetResolvePrerequesites.cmake new file mode 100644 index 00000000000..2ae34ff1c87 --- /dev/null +++ b/cmake/TargetResolvePrerequesites.cmake @@ -0,0 +1,163 @@ +set(THIS_FILE ${CMAKE_CURRENT_LIST_FILE}) + +include(CMakeParseArguments) + +function(caffe_prerequisites_directories VAR) + if(BUILD_SHARED_LIBS) + # Append the caffe library output directory + list(APPEND _directories $) + endif() + # Add boost to search directories + list(APPEND _directories ${Boost_LIBRARY_DIRS}) + # Add gflags to search directories + # gflags_DIR should point to root/CMake + get_filename_component(_dir ${gflags_DIR} DIRECTORY) + list(APPEND _directories ${_dir}/lib) + # Add glog to search directories + # glog_DIR should point to root/lib/cmake/glog + get_filename_component(_dir ${glog_DIR} DIRECTORY) + get_filename_component(_dir ${_dir} DIRECTORY) + get_filename_component(_dir ${_dir} DIRECTORY) + list(APPEND _directories ${_dir}/bin) + # Add HDF5 to search directories + # HDF5_DIR should point to root/CMake + get_filename_component(_dir ${HDF5_DIR} DIRECTORY) + list(APPEND _directories ${_dir}/bin) + # Add OpenCV to search directories + get_filename_component(_dir ${OpenCV_LIB_PATH} DIRECTORY) + list(APPEND _directories ${_dir}/bin) + if(CUDNN_FOUND AND HAVE_CUDNN) + # Add OpenCV to search directories + get_filename_component(_dir ${CUDNN_LIBRARY} DIRECTORY) + get_filename_component(_dir ${_dir} DIRECTORY) + get_filename_component(_dir ${_dir} DIRECTORY) + list(APPEND _directories ${_dir}/bin) + endif() + list(REMOVE_DUPLICATES _directories) + set(${VAR} ${_directories} PARENT_SCOPE) +endfunction() + +function(caffe_copy_prerequisites target) + caffe_prerequisites_directories(_directories) + target_copy_prerequisites(${target} ${ARGN} DIRECTORIES ${_directories}) +endfunction() + +function(caffe_install_prerequisites target) + caffe_prerequisites_directories(_directories) + target_install_prerequisites(${target} ${ARGN} DIRECTORIES ${_directories}) +endfunction() + +function(target_copy_prerequisites target) + set(options USE_HARD_LINKS) + set(oneValueArgs DESTINATION) + set(multiValueArgs DIRECTORIES) + cmake_parse_arguments(tcp "${options}" "${oneValueArgs}" + "${multiValueArgs}" ${ARGN}) + if(NOT tcp_DESTINATION) + set(tcp_DESTINATION $) + endif() + string(REPLACE ";" "@@" tcp_DIRECTORIES "${tcp_DIRECTORIES}") + add_custom_command(TARGET ${target} POST_BUILD + COMMAND ${CMAKE_COMMAND} + -DTARGET=$ + -DDESTINATION=${tcp_DESTINATION} + -DUSE_HARD_LINKS=${tcp_USE_HARD_LINKS} + -DDIRECTORIES=${tcp_DIRECTORIES} + -P ${THIS_FILE} + ) +endfunction() + +function(target_install_prerequisites target) + set(options ) + set(oneValueArgs DESTINATION) + set(multiValueArgs DIRECTORIES) + cmake_parse_arguments(tcp "${options}" "${oneValueArgs}" + "${multiValueArgs}" ${ARGN}) + if(NOT tcp_DESTINATION) + set(tcp_DESTINATION bin) + endif() + if(NOT IS_ABSOLUTE ${tcp_DESTINATION}) + set(tcp_DESTINATION ${CMAKE_INSTALL_PREFIX}/${tcp_DESTINATION}) + endif() + string(REPLACE ";" "@@" tcp_DIRECTORIES "${tcp_DIRECTORIES}") + set(_command_output ${CMAKE_CURRENT_BINARY_DIR}/${target}-install-prerequisites.stamp) + add_custom_command(OUTPUT ${_command_output} + COMMAND ${CMAKE_COMMAND} + -DTARGET=$ + -DDESTINATION=${tcp_DESTINATION} + -DUSE_HARD_LINKS=0 + -DDIRECTORIES=${tcp_DIRECTORIES} + -P ${THIS_FILE} + COMMAND ${CMAKE_COMMAND} -E touch ${_command_output} + ) + add_custom_target(${target}_install_prerequisites ALL + DEPENDS ${_command_output}) + install(FILES ${_command_output} DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/tmp) +endfunction() + +function(create_hardlink link target result_variable) + file(TO_NATIVE_PATH ${link} _link) + file(TO_NATIVE_PATH ${target} _target) + execute_process(COMMAND cmd /c mklink /H "${_link}" "${_target}" + RESULT_VARIABLE _result + OUTPUT_VARIABLE _stdout + ERROR_VARIABLE _stderr + ) + set(${result_variable} ${_result} PARENT_SCOPE) +endfunction() + +function(copy_changed_file filename destination use_hard_links) + set(_copy 1) + set(_src_name ${filename}) + get_filename_component(_name ${_src_name} NAME) + set(_dst_name ${destination}/${_name}) + + # lock a file to ensure that no two cmake processes + # try to copy the same file at the same time in parallel + # builds + string(SHA1 _hash ${_dst_name}) + set(_lock_file ${CMAKE_BINARY_DIR}/${_hash}.lock) + file(LOCK ${_lock_file} GUARD FUNCTION) + + if(EXISTS ${_dst_name}) + file(TIMESTAMP ${_dst_name} _dst_time) + file(TIMESTAMP ${_src_name} _src_time) + if(${_dst_time} STREQUAL ${_src_time}) + # skip this library if the destination and source + # have the same time stamp + return() + else() + # file has changed remove + file(REMOVE ${_dst_name}) + endif() + endif() + + if(use_hard_links) + message(STATUS "Creating hardlink for ${_name} in ${destination}") + create_hardlink(${_dst_name} ${_src_name} _result) + if(_result EQUAL 0) + set(_copy 0) + else() + message(STATUS "Failed to create hardlink ${_dst_name}. Copying instead.") + endif() + endif() + if(_copy) + message(STATUS "Copying ${_name} to ${destination}") + file(COPY ${_src_name} DESTINATION ${DESTINATION}) + endif() +endfunction() + + +if(CMAKE_SCRIPT_MODE_FILE) + include(GetPrerequisites) + # Recreate a list by replacing the @@ with ; + string(REPLACE "@@" ";" DIRECTORIES "${DIRECTORIES}") + # Get a recursive list of dependencies required by target using dumpbin + get_prerequisites(${TARGET} _prerequisites 1 1 "" "${DIRECTORIES}") + foreach(_prereq ${_prerequisites}) + # Resolve the dependency using the list of directories + gp_resolve_item("${TARGET}" "${_prereq}" "" "${DIRECTORIES}" resolved_file) + # Copy or create hardlink (if possible) + copy_changed_file(${resolved_file} ${DESTINATION} ${USE_HARD_LINKS}) + endforeach() +endif() \ No newline at end of file diff --git a/cmake/WindowsDownloadPrebuiltDependencies.cmake b/cmake/WindowsDownloadPrebuiltDependencies.cmake new file mode 100644 index 00000000000..f48c4ee5b90 --- /dev/null +++ b/cmake/WindowsDownloadPrebuiltDependencies.cmake @@ -0,0 +1,75 @@ +set(DEPENDENCIES_URL_1800_27 "https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v120_x64_py27_1.0.1.tar.bz2") +set(DEPENDENCIES_SHA_1800_27 "3f45fe3f27b27a7809f9de1bd85e56888b01dbe2") +set(DEPENDENCIES_URL_1900_27 "https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v140_x64_py27_1.0.1.tar.bz2") +set(DEPENDENCIES_SHA_1900_27 "427faf33745cf8cd70c7d043c85db7dda7243122") +set(DEPENDENCIES_URL_1900_35 "https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v140_x64_py35_1.0.1.tar.bz2") +set(DEPENDENCIES_SHA_1900_35 "1f55dac54aeab7ae3a1cda145ca272dea606bdf9") + +caffe_option(USE_PREBUILT_DEPENDENCIES "Download and use the prebuilt dependencies" ON IF MSVC) +if(MSVC) + set(CAFFE_DEPENDENCIES_DOWNLOAD_DIR ${CMAKE_CURRENT_BINARY_DIR} CACHE PATH "Download directory for prebuilt dependencies") + set(CAFFE_DEPENDENCIES_DIR ${CMAKE_CURRENT_BINARY_DIR}) +endif() +if(USE_PREBUILT_DEPENDENCIES) + # Determine the python version + if(BUILD_python) + if(NOT PYTHONINTERP_FOUND) + if(NOT "${python_version}" VERSION_LESS "3.0.0") + find_package(PythonInterp 3.5) + else() + find_package(PythonInterp 2.7) + endif() + endif() + set(_pyver ${PYTHON_VERSION_MAJOR}${PYTHON_VERSION_MINOR}) + else() + message(STATUS "Building without python. Prebuilt dependencies will default to Python 2.7") + set(_pyver 27) + endif() + if(NOT DEFINED DEPENDENCIES_URL_${MSVC_VERSION}_${_pyver}) + message(FATAL_ERROR "Could not find url for MSVC version = ${MSVC_VERSION} and Python version = {PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR}.") + endif() + # set the dependencies URL and SHA1 + set(DEPENDENCIES_URL ${DEPENDENCIES_URL_${MSVC_VERSION}_${_pyver}}) + set(DEPENDENCIES_SHA ${DEPENDENCIES_SHA_${MSVC_VERSION}_${_pyver}}) + # create the download directory if it does not exist + if(NOT EXISTS ${CAFFE_DEPENDENCIES_DOWNLOAD_DIR}) + file(MAKE_DIRECTORY ${CAFFE_DEPENDENCIES_DOWNLOAD_DIR}) + endif() + # download and extract the file if it does not exist or if does not match the sha1 + get_filename_component(_download_filename ${DEPENDENCIES_URL} NAME) + set(_download_path ${CAFFE_DEPENDENCIES_DOWNLOAD_DIR}/${_download_filename}) + set(_download_file 1) + if(EXISTS ${_download_path}) + file(SHA1 ${_download_path} _file_sha) + if("${_file_sha}" STREQUAL "${DEPENDENCIES_SHA}") + set(_download_file 0) + else() + set(_download_file 1) + message(STATUS "Removing file because sha1 does not match.") + file(REMOVE ${_download_path}) + endif() + endif() + if(_download_file) + message(STATUS "Downloading file dependencies") + file(DOWNLOAD "${DEPENDENCIES_URL}" + "${_download_path}" + EXPECTED_HASH SHA1=${DEPENDENCIES_SHA} + SHOW_PROGRESS 1 + ) + if(EXISTS ${CAFFE_DEPENDENCIES_DIR}/libraries) + file(REMOVE_RECURSE ${CAFFE_DEPENDENCIES_DIR}/libraries) + endif() + endif() + if(EXISTS ${_download_path} AND NOT EXISTS ${CAFFE_DEPENDENCIES_DIR}/libraries) + message(STATUS "Extracting dependencies") + execute_process(COMMAND ${CMAKE_COMMAND} -E tar xjf ${_download_path} + WORKING_DIRECTORY ${CAFFE_DEPENDENCIES_DIR} + ) + endif() + if(EXISTS ${CAFFE_DEPENDENCIES_DIR}/libraries/caffe-builder-config.cmake) + include(${CAFFE_DEPENDENCIES_DIR}/libraries/caffe-builder-config.cmake) + else() + message(FATAL_ERROR "Something went wrong while dowloading dependencies could not open caffe-builder-config.cmake") + endif() +endif() + diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt index 5d7f3d7c03a..9fcb6728874 100644 --- a/python/CMakeLists.txt +++ b/python/CMakeLists.txt @@ -27,10 +27,14 @@ elseif(WIN32) COMMAND ${CMAKE_COMMAND} -E copy_if_different $ ${PROJECT_SOURCE_DIR}/python/caffe COMMAND ${CMAKE_COMMAND} -E make_directory ${PROJECT_SOURCE_DIR}/python/caffe/proto COMMAND ${CMAKE_COMMAND} -E touch ${PROJECT_SOURCE_DIR}/python/caffe/proto/__init__.py - COMMAND (robocopy "\"${proto_gen_folder}\" \"${PROJECT_SOURCE_DIR}/python/caffe/proto\" *.py") ^& IF %ERRORLEVEL% LEQ 4 exit /B 0 + COMMAND (robocopy "\"${proto_gen_folder}\" \"${PROJECT_SOURCE_DIR}/python/caffe/proto\" *.py") ^& IF %ERRORLEVEL% LEQ 4 set ERRORLEVEL=0 COMMENT "Creating symlink ${__linkname} -> ${PROJECT_BINARY_DIR}/lib/_caffe.pyd") endif() +if(MSVC AND COPY_PREREQUISITES) + caffe_copy_prerequisites(pycaffe DESTINATION ${PROJECT_SOURCE_DIR}/python/caffe USE_HARD_LINKS) +endif() + # ---[ Install # scripts file(GLOB python_files *.py requirements.txt) @@ -48,3 +52,8 @@ install(DIRECTORY caffe # _caffe.so install(TARGETS pycaffe DESTINATION python/caffe) +if(MSVC AND INSTALL_PREREQUISITES) + caffe_install_prerequisites(pycaffe DESTINATION python/caffe) +endif() + + diff --git a/scripts/build_win.cmd b/scripts/build_win.cmd index d408e7048d1..cc541e0d493 100644 --- a/scripts/build_win.cmd +++ b/scripts/build_win.cmd @@ -9,6 +9,7 @@ if DEFINED APPVEYOR ( if NOT DEFINED CPU_ONLY set CPU_ONLY=1 if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0 + if NOT DEFINED PYTHON_VERSION set PYTHON_VERSION=2 if NOT DEFINED BUILD_PYTHON set BUILD_PYTHON=1 if NOT DEFINED BUILD_PYTHON_LAYER set BUILD_PYTHON_LAYER=1 if NOT DEFINED BUILD_MATLAB set BUILD_MATLAB=0 @@ -18,7 +19,15 @@ if DEFINED APPVEYOR ( if NOT DEFINED RUN_INSTALL set RUN_INSTALL=1 :: Set python 2.7 with conda as the default python - set PATH=C:\Miniconda-x64;C:\Miniconda-x64\Scripts;C:\Miniconda-x64\Library\bin;!PATH! + if !PYTHON_VERSION! EQU 2 ( + set CONDA_ROOT=C:\Miniconda-x64 + ) + :: Set python 3.5 with conda as the default python + if !PYTHON_VERSION! EQU 3 ( + set CONDA_ROOT=C:\Miniconda35-x64 + ) + set PATH=!CONDA_ROOT!;!CONDA_ROOT!\Scripts;!CONDA_ROOT!\Library\bin;!PATH! + :: Check that we have the right python version !PYTHON_EXE! --version :: Add the required channels @@ -26,9 +35,8 @@ if DEFINED APPVEYOR ( conda config --add channels willyd :: Update conda conda update conda -y - :: Create an environment - :: Todo create protobuf package for vc14 - conda install --yes cmake ninja numpy scipy protobuf==3.1.0.vc12 six scikit-image pyyaml + :: Download other required packages + conda install --yes cmake ninja numpy scipy protobuf==3.1.0 six scikit-image pyyaml if ERRORLEVEL 1 ( echo ERROR: Conda update or install failed @@ -41,30 +49,37 @@ if DEFINED APPVEYOR ( set RUN_TESTS=0 ) + :: Disable linting with python 3 until we find why the script fails + if !PYTHON_VERSION! EQU 3 ( + set RUN_LINT=0 + ) + ) else ( :: Change the settings here to match your setup :: Change MSVC_VERSION to 12 to use VS 2013 - if NOT DEFINED MSVC_VERSION set MSVC_VERSION=14 + set MSVC_VERSION=14 :: Change to 1 to use Ninja generator (builds much faster) - if NOT DEFINED WITH_NINJA set WITH_NINJA=0 + set WITH_NINJA=1 :: Change to 1 to build caffe without CUDA support - if NOT DEFINED CPU_ONLY set CPU_ONLY=0 + set CPU_ONLY=0 :: Change to Debug to build Debug. This is only relevant for the Ninja generator the Visual Studio generator will generate both Debug and Release configs - if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release + set CMAKE_CONFIG=Release :: Change to 1 to build a caffe.dll - if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0 + set CMAKE_BUILD_SHARED_LIBS=0 + :: Change to 3 if using python 3.5 (only 2.7 and 3.5 are supported) + set PYTHON_VERSION=2 :: Change these options for your needs. - if NOT DEFINED BUILD_PYTHON set BUILD_PYTHON=1 - if NOT DEFINED BUILD_PYTHON_LAYER set BUILD_PYTHON_LAYER=1 - if NOT DEFINED BUILD_MATLAB set BUILD_MATLAB=0 + set BUILD_PYTHON=1 + set BUILD_PYTHON_LAYER=1 + set BUILD_MATLAB=0 :: If python is on your path leave this alone - if NOT DEFINED PYTHON_EXE set PYTHON_EXE=python + set PYTHON_EXE=python :: Run the tests - if NOT DEFINED RUN_TESTS set RUN_TESTS=0 + set RUN_TESTS=0 :: Run lint - if NOT DEFINED RUN_LINT set RUN_LINT=0 + set RUN_LINT=0 :: Build the install target - if NOT DEFINED RUN_INSTALL set RUN_INSTALL=0 + set RUN_INSTALL=0 ) :: Set the appropriate CMake generator @@ -94,6 +109,7 @@ echo INFO: CMAKE_GENERATOR = "!CMAKE_GENERATOR!" echo INFO: CPU_ONLY = !CPU_ONLY! echo INFO: CMAKE_CONFIG = !CMAKE_CONFIG! echo INFO: CMAKE_BUILD_SHARED_LIBS = !CMAKE_BUILD_SHARED_LIBS! +echo INFO: PYTHON_VERSION = !PYTHON_VERSION! echo INFO: BUILD_PYTHON = !BUILD_PYTHON! echo INFO: BUILD_PYTHON_LAYER = !BUILD_PYTHON_LAYER! echo INFO: BUILD_MATLAB = !BUILD_MATLAB! @@ -112,30 +128,9 @@ if !RUN_TESTS! EQU 1 ( ) ) -:: Create build directory and configure cmake -if EXIST build ( - echo ERROR: build directory already exists in %cd%\build please remove it and start over. - exit /b 1 -) - -mkdir build +if NOT EXIST build mkdir build pushd build -:: Download dependencies from VS x64 -echo INFO: Downloading dependencies -"%PYTHON_EXE%" "%~dp0\download_prebuilt_dependencies.py" --msvc_version v%MSVC_VERSION%0 - -if ERRORLEVEL 1 ( - echo ERROR: Downloading dependencies failed - exit /b 1 -) - - -:: Add the dependencies to the PATH -if EXIST "%cd%\libraries\prependpath.bat" ( - call "%cd%\libraries\prependpath.bat" -) - :: Setup the environement for VS x64 set batch_file=!VS%MSVC_VERSION%0COMNTOOLS!..\..\VC\vcvarsall.bat call "%batch_file%" amd64 @@ -151,7 +146,8 @@ cmake -G"!CMAKE_GENERATOR!" ^ -DBUILD_python_layer:BOOL=%BUILD_PYTHON_LAYER% ^ -DBUILD_matlab:BOOL=%BUILD_MATLAB% ^ -DCPU_ONLY:BOOL=%CPU_ONLY% ^ - -C "%cd%\libraries\caffe-builder-config.cmake" ^ + -DCOPY_PREREQUISITES:BOOL=1 ^ + -DINSTALL_PREREQUISITES:BOOL=1 ^ "%~dp0\.." if ERRORLEVEL 1 ( diff --git a/src/caffe/test/CMakeLists.txt b/src/caffe/test/CMakeLists.txt index a918b609178..4a6ad64655f 100644 --- a/src/caffe/test/CMakeLists.txt +++ b/src/caffe/test/CMakeLists.txt @@ -31,6 +31,10 @@ target_link_libraries(${the_target} gtest ${Caffe_LINK}) caffe_default_properties(${the_target}) caffe_set_runtime_directory(${the_target} "${PROJECT_BINARY_DIR}/test") +if(MSVC AND COPY_PREREQUISITES) + caffe_copy_prerequisites(${the_target} USE_HARD_LINKS) +endif() + # ---[ Adding runtest add_custom_target(runtest COMMAND ${the_target} ${test_args} WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}) \ No newline at end of file diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt index 197d31a746b..488b182b1fc 100644 --- a/tools/CMakeLists.txt +++ b/tools/CMakeLists.txt @@ -15,6 +15,10 @@ foreach(source ${srcs}) target_link_libraries(${name} ${Caffe_LINK}) caffe_default_properties(${name}) + if(MSVC AND COPY_PREREQUISITES) + caffe_copy_prerequisites(${name} USE_HARD_LINKS) + endif() + # set back RUNTIME_OUTPUT_DIRECTORY caffe_set_runtime_directory(${name} "${PROJECT_BINARY_DIR}/tools") caffe_set_solution_folder(${name} tools) @@ -31,4 +35,8 @@ foreach(source ${srcs}) # Install install(TARGETS ${name} DESTINATION bin) + + if(MSVC AND INSTALL_PREREQUISITES) + caffe_install_prerequisites(${name} DESTINATION bin) + endif() endforeach(source) From f4db952dad204b78cd0d636e09a69bc04ad92f78 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Wed, 18 Jan 2017 09:01:53 -0500 Subject: [PATCH 31/51] Fixed AppVeyor artifacts download links --- README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index a1dfc992624..26a1270cb6a 100644 --- a/README.md +++ b/README.md @@ -12,11 +12,11 @@ This branch of Caffe ports the framework to Windows. Prebuilt binaries can be downloaded from the latest CI build on appveyor for the following configurations: -- Visual Studio 2015, CPU only, Python 3.5: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D3), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DDebug%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D3) +- Visual Studio 2015, CPU only, Python 3.5: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D3), ~~[Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D3)~~ -- Visual Studio 2015, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D2), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D14%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DDebug%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D2) +- Visual Studio 2015, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2) -- Visual Studio 2013, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D12%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DRelease%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D2), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A+MSVC_VERSION%3D12%2C+WITH_NINJA%3D0%2C+CMAKE_CONFIG%3DDebug%2C+CMAKE_BUILD_SHARED_LIBS%3D0+PYTHON_VERSION%3D2) +- Visual Studio 2013, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D12%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D12%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2) ## Windows Setup From 372e920eb5c0ccdf70a7caaa344847d1485919c6 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Mon, 24 Oct 2016 22:16:26 -0400 Subject: [PATCH 32/51] Added CI with CUDA on Windows and disabled some Debug builds on AppVeyor --- README.md | 10 +++++++--- appveyor.yml | 23 +++++++++++++++++++++++ scripts/appveyor/appveyor_install_cuda.cmd | 13 +++++++++++++ scripts/build_win.cmd | 9 +++++++++ 4 files changed, 52 insertions(+), 3 deletions(-) create mode 100644 scripts/appveyor/appveyor_install_cuda.cmd diff --git a/README.md b/README.md index 26a1270cb6a..6a37ac03e80 100644 --- a/README.md +++ b/README.md @@ -12,11 +12,15 @@ This branch of Caffe ports the framework to Windows. Prebuilt binaries can be downloaded from the latest CI build on appveyor for the following configurations: -- Visual Studio 2015, CPU only, Python 3.5: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D3), ~~[Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D3)~~ +- Visual Studio 2015, CPU only, Python 3.5: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D3%2C%20WITH_CUDA%3D0), ~~[Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D3%2C%20WITH_CUDA%3D0)~~ -- Visual Studio 2015, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2) +- Visual Studio 2015, CUDA 8.0, Python 3.5: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D3%2C%20WITH_CUDA%3D1) -- Visual Studio 2013, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D12%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D12%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2) +- Visual Studio 2015, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2%2C%20WITH_CUDA%3D0), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2%2C%20WITH_CUDA%3D0) + +- Visual Studio 2015,CUDA 8.0, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2%2C%20WITH_CUDA%3D1) + +- Visual Studio 2013, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D12%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2%2C%20WITH_CUDA%3D0), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D12%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2%2C%20WITH_CUDA%3D0) ## Windows Setup diff --git a/appveyor.yml b/appveyor.yml index 5e1de384aa7..f37d4e6feb7 100644 --- a/appveyor.yml +++ b/appveyor.yml @@ -7,54 +7,77 @@ environment: CMAKE_CONFIG: Release CMAKE_BUILD_SHARED_LIBS: 0 PYTHON_VERSION: 3 + WITH_CUDA: 0 - MSVC_VERSION: 14 WITH_NINJA: 0 CMAKE_CONFIG: Release CMAKE_BUILD_SHARED_LIBS: 0 PYTHON_VERSION: 2 + WITH_CUDA: 0 - MSVC_VERSION: 14 WITH_NINJA: 0 CMAKE_CONFIG: Debug CMAKE_BUILD_SHARED_LIBS: 0 PYTHON_VERSION: 2 + WITH_CUDA: 0 - MSVC_VERSION: 14 WITH_NINJA: 1 CMAKE_CONFIG: Release CMAKE_BUILD_SHARED_LIBS: 0 PYTHON_VERSION: 2 + WITH_CUDA: 0 - MSVC_VERSION: 14 WITH_NINJA: 1 CMAKE_CONFIG: Debug CMAKE_BUILD_SHARED_LIBS: 0 PYTHON_VERSION: 2 + WITH_CUDA: 0 + + - MSVC_VERSION: 14 + WITH_NINJA: 1 + CMAKE_CONFIG: Release + CMAKE_BUILD_SHARED_LIBS: 0 + PYTHON_VERSION: 2 + WITH_CUDA: 1 + + - MSVC_VERSION: 14 + WITH_NINJA: 1 + CMAKE_CONFIG: Release + CMAKE_BUILD_SHARED_LIBS: 0 + PYTHON_VERSION: 3 + WITH_CUDA: 1 - MSVC_VERSION: 12 WITH_NINJA: 0 CMAKE_CONFIG: Release CMAKE_BUILD_SHARED_LIBS: 0 PYTHON_VERSION: 2 + WITH_CUDA: 0 - MSVC_VERSION: 12 WITH_NINJA: 0 CMAKE_CONFIG: Debug CMAKE_BUILD_SHARED_LIBS: 0 PYTHON_VERSION: 2 + WITH_CUDA: 0 - MSVC_VERSION: 12 WITH_NINJA: 1 CMAKE_CONFIG: Release CMAKE_BUILD_SHARED_LIBS: 0 PYTHON_VERSION: 2 + WITH_CUDA: 0 - MSVC_VERSION: 12 WITH_NINJA: 1 CMAKE_CONFIG: Debug CMAKE_BUILD_SHARED_LIBS: 0 PYTHON_VERSION: 2 + WITH_CUDA: 0 build_script: - cmd: >- diff --git a/scripts/appveyor/appveyor_install_cuda.cmd b/scripts/appveyor/appveyor_install_cuda.cmd new file mode 100644 index 00000000000..c8f6c1b32df --- /dev/null +++ b/scripts/appveyor/appveyor_install_cuda.cmd @@ -0,0 +1,13 @@ +@echo off +echo Downloading CUDA toolkit 8 ... +appveyor DownloadFile https://developer.nvidia.com/compute/cuda/8.0/prod/local_installers/cuda_8.0.44_windows-exe -FileName cuda_8.0.44_windows.exe +echo Installing CUDA toolkit 8 ... +cuda_8.0.44_windows.exe -s compiler_8.0 ^ + cublas_8.0 ^ + cublas_dev_8.0 ^ + cudart_8.0 ^ + curand_8.0 ^ + curand_dev_8.0 +:: Add CUDA toolkit to PATH +set PATH=%ProgramFiles%\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin;%ProgramFiles%\NVIDIA GPU Computing Toolkit\CUDA\v8.0\libnvvp;%PATH% +nvcc -V \ No newline at end of file diff --git a/scripts/build_win.cmd b/scripts/build_win.cmd index cc541e0d493..cb98f40c09b 100644 --- a/scripts/build_win.cmd +++ b/scripts/build_win.cmd @@ -43,6 +43,15 @@ if DEFINED APPVEYOR ( exit /b 1 ) + :: Install cuda and disable tests if needed + if %WITH_CUDA% == 1 ( + call %~dp0\appveyor\appveyor_install_cuda.cmd + set CPU_ONLY=0 + set RUN_TESTS=0 + ) else ( + set CPU_ONLY=1 + ) + :: Disable the tests in debug config if "%CMAKE_CONFIG%" == "Debug" ( echo Disabling tests on appveyor with config == %CMAKE_CONFIG% From a3f1cce1764145455bd5fc1cdf200e5f469848b4 Mon Sep 17 00:00:00 2001 From: Fredrik Orderud Date: Fri, 20 Jan 2017 11:09:58 +0100 Subject: [PATCH 33/51] Fix the following error when running the script without defining "WITH_CUDA": "( was unexpected at this time." --- scripts/build_win.cmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/build_win.cmd b/scripts/build_win.cmd index cb98f40c09b..0403f29a64c 100644 --- a/scripts/build_win.cmd +++ b/scripts/build_win.cmd @@ -44,7 +44,7 @@ if DEFINED APPVEYOR ( ) :: Install cuda and disable tests if needed - if %WITH_CUDA% == 1 ( + if !WITH_CUDA! == 1 ( call %~dp0\appveyor\appveyor_install_cuda.cmd set CPU_ONLY=0 set RUN_TESTS=0 From 8a49d455c3589d4616a6f6191d6226353d13421e Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Fri, 20 Jan 2017 08:44:58 -0500 Subject: [PATCH 34/51] Fix broken download links for CUDA enabled AppVeyor builds --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 6a37ac03e80..26b779a347a 100644 --- a/README.md +++ b/README.md @@ -14,11 +14,11 @@ Prebuilt binaries can be downloaded from the latest CI build on appveyor for the - Visual Studio 2015, CPU only, Python 3.5: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D3%2C%20WITH_CUDA%3D0), ~~[Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D3%2C%20WITH_CUDA%3D0)~~ -- Visual Studio 2015, CUDA 8.0, Python 3.5: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D3%2C%20WITH_CUDA%3D1) +- Visual Studio 2015, CUDA 8.0, Python 3.5: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D1%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D3%2C%20WITH_CUDA%3D1) - Visual Studio 2015, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2%2C%20WITH_CUDA%3D0), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2%2C%20WITH_CUDA%3D0) -- Visual Studio 2015,CUDA 8.0, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2%2C%20WITH_CUDA%3D1) +- Visual Studio 2015,CUDA 8.0, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D14%2C%20WITH_NINJA%3D1%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2%2C%20WITH_CUDA%3D1) - Visual Studio 2013, CPU only, Python 2.7: [Caffe Release](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D12%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DRelease%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2%2C%20WITH_CUDA%3D0), [Caffe Debug](https://ci.appveyor.com/api/projects/BVLC/caffe/artifacts/build/caffe.zip?branch=windows&job=Environment%3A%20MSVC_VERSION%3D12%2C%20WITH_NINJA%3D0%2C%20CMAKE_CONFIG%3DDebug%2C%20CMAKE_BUILD_SHARED_LIBS%3D0%2C%20PYTHON_VERSION%3D2%2C%20WITH_CUDA%3D0) From 46b3d46127b6065d5ea19c268e2e43aca3b848a8 Mon Sep 17 00:00:00 2001 From: Fredrik Orderud Date: Fri, 20 Jan 2017 14:51:43 +0100 Subject: [PATCH 35/51] Make it possible to change build configuration on Windows without having to edit build_win.cmd. --- scripts/build_win.cmd | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/scripts/build_win.cmd b/scripts/build_win.cmd index 0403f29a64c..67b498b21d8 100644 --- a/scripts/build_win.cmd +++ b/scripts/build_win.cmd @@ -66,29 +66,29 @@ if DEFINED APPVEYOR ( ) else ( :: Change the settings here to match your setup :: Change MSVC_VERSION to 12 to use VS 2013 - set MSVC_VERSION=14 + if NOT DEFINED MSVC_VERSION set MSVC_VERSION=14 :: Change to 1 to use Ninja generator (builds much faster) - set WITH_NINJA=1 + if NOT DEFINED WITH_NINJA set WITH_NINJA=1 :: Change to 1 to build caffe without CUDA support - set CPU_ONLY=0 + if NOT DEFINED CPU_ONLY set CPU_ONLY=0 :: Change to Debug to build Debug. This is only relevant for the Ninja generator the Visual Studio generator will generate both Debug and Release configs - set CMAKE_CONFIG=Release + if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release :: Change to 1 to build a caffe.dll - set CMAKE_BUILD_SHARED_LIBS=0 + if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0 :: Change to 3 if using python 3.5 (only 2.7 and 3.5 are supported) - set PYTHON_VERSION=2 + if NOT DEFINED PYTHON_VERSION set PYTHON_VERSION=2 :: Change these options for your needs. - set BUILD_PYTHON=1 - set BUILD_PYTHON_LAYER=1 - set BUILD_MATLAB=0 + if NOT DEFINED BUILD_PYTHON set BUILD_PYTHON=1 + if NOT DEFINED BUILD_PYTHON_LAYER set BUILD_PYTHON_LAYER=1 + if NOT DEFINED BUILD_MATLAB set BUILD_MATLAB=0 :: If python is on your path leave this alone - set PYTHON_EXE=python + if NOT DEFINED PYTHON_EXE set PYTHON_EXE=python :: Run the tests - set RUN_TESTS=0 + if NOT DEFINED RUN_TESTS set RUN_TESTS=0 :: Run lint - set RUN_LINT=0 + if NOT DEFINED RUN_LINT set RUN_LINT=0 :: Build the install target - set RUN_INSTALL=0 + if NOT DEFINED RUN_INSTALL set RUN_INSTALL=0 ) :: Set the appropriate CMake generator From d14012ea977c785949786d50cd44356ec4f53f55 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Fri, 3 Feb 2017 08:51:00 -0500 Subject: [PATCH 36/51] Removed unnecessary 1 with SHOW_PROGRESS option and fixed error message in WindowsDownloadPrebuiltDependencies.cmake --- cmake/WindowsDownloadPrebuiltDependencies.cmake | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cmake/WindowsDownloadPrebuiltDependencies.cmake b/cmake/WindowsDownloadPrebuiltDependencies.cmake index f48c4ee5b90..3fe2fabaf93 100644 --- a/cmake/WindowsDownloadPrebuiltDependencies.cmake +++ b/cmake/WindowsDownloadPrebuiltDependencies.cmake @@ -26,7 +26,7 @@ if(USE_PREBUILT_DEPENDENCIES) set(_pyver 27) endif() if(NOT DEFINED DEPENDENCIES_URL_${MSVC_VERSION}_${_pyver}) - message(FATAL_ERROR "Could not find url for MSVC version = ${MSVC_VERSION} and Python version = {PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR}.") + message(FATAL_ERROR "Could not find url for MSVC version = ${MSVC_VERSION} and Python version = ${PYTHON_VERSION_MAJOR}.${PYTHON_VERSION_MINOR}.") endif() # set the dependencies URL and SHA1 set(DEPENDENCIES_URL ${DEPENDENCIES_URL_${MSVC_VERSION}_${_pyver}}) @@ -54,7 +54,7 @@ if(USE_PREBUILT_DEPENDENCIES) file(DOWNLOAD "${DEPENDENCIES_URL}" "${_download_path}" EXPECTED_HASH SHA1=${DEPENDENCIES_SHA} - SHOW_PROGRESS 1 + SHOW_PROGRESS ) if(EXISTS ${CAFFE_DEPENDENCIES_DIR}/libraries) file(REMOVE_RECURSE ${CAFFE_DEPENDENCIES_DIR}/libraries) From 2106a6234188a374caf06dd92397c349bb5c26c2 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Thu, 16 Feb 2017 22:42:05 -0500 Subject: [PATCH 37/51] Added nccl ExternalProject to build nccl on Windows --- cmake/Dependencies.cmake | 2 +- cmake/External/nccl.cmake | 35 ++++++++++++++++++++++++++++++ cmake/TargetResolvePrerequesites.cmake | 29 +++++++++++++++++++++++-- python/caffe/_caffe.cpp | 12 +++++++--- scripts/appveyor/appveyor_install_cuda.cmd | 3 ++- scripts/build_win.cmd | 6 +++++ 6 files changed, 80 insertions(+), 7 deletions(-) create mode 100644 cmake/External/nccl.cmake diff --git a/cmake/Dependencies.cmake b/cmake/Dependencies.cmake index 2cde0a69ddd..2ba2fe64842 100644 --- a/cmake/Dependencies.cmake +++ b/cmake/Dependencies.cmake @@ -85,7 +85,7 @@ if(NOT HAVE_CUDA) endif() if(USE_NCCL) - find_package(NCCL REQUIRED) + include("cmake/External/nccl.cmake") include_directories(SYSTEM ${NCCL_INCLUDE_DIR}) list(APPEND Caffe_LINKER_LIBS ${NCCL_LIBRARIES}) add_definitions(-DUSE_NCCL) diff --git a/cmake/External/nccl.cmake b/cmake/External/nccl.cmake new file mode 100644 index 00000000000..5f97f292239 --- /dev/null +++ b/cmake/External/nccl.cmake @@ -0,0 +1,35 @@ +# if (NOT __NCCL_INCLUDED) # guard against multiple includes + set(__NCCL_INCLUDED TRUE) + if(MSVC) + # use the system-wide nccl if present + find_package(NCCL) + if (NCCL_FOUND) + set(NCCL_EXTERNAL FALSE) + else() + # build directory + set(nccl_PREFIX ${CMAKE_BINARY_DIR}/external/nccl-prefix) + # install directory + set(nccl_INSTALL ${CMAKE_BINARY_DIR}/external/nccl-install) + ExternalProject_Add(nccl + PREFIX ${nccl_PREFIX} + URL https://github.com/willyd/nccl/archive/470b3130457f125f4608c7baee71123aa16a3b12.zip + UPDATE_COMMAND "" + INSTALL_DIR ${nccl_INSTALL} + CMAKE_ARGS -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} + -DCMAKE_INSTALL_PREFIX=${nccl_INSTALL} + -DBUILD_SHARED_LIBS=OFF + -DNCCL_BUILD_TESTS:BOOL=OFF + + LOG_DOWNLOAD 1 + LOG_INSTALL 1 + BUILD_BYPRODUCTS ${nccl_INSTALL}/include ${nccl_INSTALL}/lib/nccl.lib + ) + + set(NCCL_INCLUDE_DIR ${nccl_INSTALL}/include) + set(NCCL_LIBRARIES ${nccl_INSTALL}/lib/nccl.lib) + endif() + else() + # default to find package on UNIX systems + find_package(NCCL REQUIRED) + endif() +# endif() \ No newline at end of file diff --git a/cmake/TargetResolvePrerequesites.cmake b/cmake/TargetResolvePrerequesites.cmake index 2ae34ff1c87..f58333f7bea 100644 --- a/cmake/TargetResolvePrerequesites.cmake +++ b/cmake/TargetResolvePrerequesites.cmake @@ -33,6 +33,14 @@ function(caffe_prerequisites_directories VAR) get_filename_component(_dir ${_dir} DIRECTORY) list(APPEND _directories ${_dir}/bin) endif() + if(USE_NCCL) + # add the nvml.dll path if we are using nccl + file(TO_CMAKE_PATH "$ENV{NVTOOLSEXT_PATH}" _nvtools_ext) + if(NOT "${_nvtools_ext}" STREQUAL "") + get_filename_component(_nvsmi_path ${_nvtools_ext}/../nvsmi ABSOLUTE) + list(APPEND _directories ${_nvsmi_path}) + endif() + endif() list(REMOVE_DUPLICATES _directories) set(${VAR} ${_directories} PARENT_SCOPE) endfunction() @@ -57,12 +65,19 @@ function(target_copy_prerequisites target) set(tcp_DESTINATION $) endif() string(REPLACE ";" "@@" tcp_DIRECTORIES "${tcp_DIRECTORIES}") + if(USE_NCCL) + # nccl loads the nvml.dll dynamically so we need + # to list it explicitely + list(APPEND _plugins nvml.dll) + endif() + string(REPLACE ";" "@@" _plugins "${_plugins}") add_custom_command(TARGET ${target} POST_BUILD COMMAND ${CMAKE_COMMAND} -DTARGET=$ -DDESTINATION=${tcp_DESTINATION} -DUSE_HARD_LINKS=${tcp_USE_HARD_LINKS} -DDIRECTORIES=${tcp_DIRECTORIES} + -DPLUGINS=${_plugins} -P ${THIS_FILE} ) endfunction() @@ -80,6 +95,12 @@ function(target_install_prerequisites target) set(tcp_DESTINATION ${CMAKE_INSTALL_PREFIX}/${tcp_DESTINATION}) endif() string(REPLACE ";" "@@" tcp_DIRECTORIES "${tcp_DIRECTORIES}") + if(USE_NCCL) + # nccl loads the nvml.dll dynamically so we need + # to list it explicitely + list(APPEND _plugins nvml.dll) + endif() + string(REPLACE ";" "@@" _plugins "${_plugins}") set(_command_output ${CMAKE_CURRENT_BINARY_DIR}/${target}-install-prerequisites.stamp) add_custom_command(OUTPUT ${_command_output} COMMAND ${CMAKE_COMMAND} @@ -87,6 +108,7 @@ function(target_install_prerequisites target) -DDESTINATION=${tcp_DESTINATION} -DUSE_HARD_LINKS=0 -DDIRECTORIES=${tcp_DIRECTORIES} + -DPLUGINS=${_plugins} -P ${THIS_FILE} COMMAND ${CMAKE_COMMAND} -E touch ${_command_output} ) @@ -152,12 +174,15 @@ if(CMAKE_SCRIPT_MODE_FILE) include(GetPrerequisites) # Recreate a list by replacing the @@ with ; string(REPLACE "@@" ";" DIRECTORIES "${DIRECTORIES}") + string(REPLACE "@@" ";" PLUGINS "${PLUGINS}") # Get a recursive list of dependencies required by target using dumpbin get_prerequisites(${TARGET} _prerequisites 1 1 "" "${DIRECTORIES}") - foreach(_prereq ${_prerequisites}) + foreach(_prereq ${_prerequisites} ${PLUGINS}) # Resolve the dependency using the list of directories gp_resolve_item("${TARGET}" "${_prereq}" "" "${DIRECTORIES}" resolved_file) # Copy or create hardlink (if possible) - copy_changed_file(${resolved_file} ${DESTINATION} ${USE_HARD_LINKS}) + if(EXISTS ${resolved_file}) + copy_changed_file(${resolved_file} ${DESTINATION} ${USE_HARD_LINKS}) + endif() endforeach() endif() \ No newline at end of file diff --git a/python/caffe/_caffe.cpp b/python/caffe/_caffe.cpp index 62fcfe22951..4a2ddd5ecc6 100644 --- a/python/caffe/_caffe.cpp +++ b/python/caffe/_caffe.cpp @@ -44,15 +44,20 @@ #define BP_GET_POINTER(cls) \ namespace boost { \ template <> \ -caffe::cls const volatile * \ -get_pointer( \ - class caffe::cls const volatile *c) { \ +const volatile caffe::cls * \ +get_pointer(const volatile caffe::cls *c) { \ return c; \ } \ } #define BP_GET_POINTER_T(cls, dtype) BP_GET_POINTER(cls) +// forward declare the NCCL class +// in case we are not using NCCL +namespace caffe { +template class NCCL; +} + BP_GET_POINTER_T(Net, float); BP_GET_POINTER_T(Layer, float); BP_GET_POINTER_T(Solver, float); @@ -62,6 +67,7 @@ BP_GET_POINTER_T(AdaGradSolver, float); BP_GET_POINTER_T(RMSPropSolver, float); BP_GET_POINTER_T(AdaDeltaSolver, float); BP_GET_POINTER_T(AdamSolver, float); +BP_GET_POINTER_T(NCCL, float); BP_GET_POINTER(Timer); #endif diff --git a/scripts/appveyor/appveyor_install_cuda.cmd b/scripts/appveyor/appveyor_install_cuda.cmd index c8f6c1b32df..722e32b22db 100644 --- a/scripts/appveyor/appveyor_install_cuda.cmd +++ b/scripts/appveyor/appveyor_install_cuda.cmd @@ -7,7 +7,8 @@ cuda_8.0.44_windows.exe -s compiler_8.0 ^ cublas_dev_8.0 ^ cudart_8.0 ^ curand_8.0 ^ - curand_dev_8.0 + curand_dev_8.0 ^ + nvml_dev_8.0 :: Add CUDA toolkit to PATH set PATH=%ProgramFiles%\NVIDIA GPU Computing Toolkit\CUDA\v8.0\bin;%ProgramFiles%\NVIDIA GPU Computing Toolkit\CUDA\v8.0\libnvvp;%PATH% nvcc -V \ No newline at end of file diff --git a/scripts/build_win.cmd b/scripts/build_win.cmd index 67b498b21d8..d7c4750e3fb 100644 --- a/scripts/build_win.cmd +++ b/scripts/build_win.cmd @@ -8,6 +8,7 @@ if DEFINED APPVEYOR ( if NOT DEFINED WITH_NINJA set WITH_NINJA=1 if NOT DEFINED CPU_ONLY set CPU_ONLY=1 if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release + if NOT DEFINED USE_NCCL set USE_NCCL=0 if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0 if NOT DEFINED PYTHON_VERSION set PYTHON_VERSION=2 if NOT DEFINED BUILD_PYTHON set BUILD_PYTHON=1 @@ -48,6 +49,7 @@ if DEFINED APPVEYOR ( call %~dp0\appveyor\appveyor_install_cuda.cmd set CPU_ONLY=0 set RUN_TESTS=0 + set USE_NCCL=1 ) else ( set CPU_ONLY=1 ) @@ -73,6 +75,8 @@ if DEFINED APPVEYOR ( if NOT DEFINED CPU_ONLY set CPU_ONLY=0 :: Change to Debug to build Debug. This is only relevant for the Ninja generator the Visual Studio generator will generate both Debug and Release configs if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release + :: Set to 1 to use NCCL + if NOT DEFINED USE_NCCL set USE_NCCL=0 :: Change to 1 to build a caffe.dll if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0 :: Change to 3 if using python 3.5 (only 2.7 and 3.5 are supported) @@ -117,6 +121,7 @@ echo INFO: WITH_NINJA = !WITH_NINJA! echo INFO: CMAKE_GENERATOR = "!CMAKE_GENERATOR!" echo INFO: CPU_ONLY = !CPU_ONLY! echo INFO: CMAKE_CONFIG = !CMAKE_CONFIG! +echo INFO: USE_NCCL = !USE_NCCL! echo INFO: CMAKE_BUILD_SHARED_LIBS = !CMAKE_BUILD_SHARED_LIBS! echo INFO: PYTHON_VERSION = !PYTHON_VERSION! echo INFO: BUILD_PYTHON = !BUILD_PYTHON! @@ -157,6 +162,7 @@ cmake -G"!CMAKE_GENERATOR!" ^ -DCPU_ONLY:BOOL=%CPU_ONLY% ^ -DCOPY_PREREQUISITES:BOOL=1 ^ -DINSTALL_PREREQUISITES:BOOL=1 ^ + -DUSE_NCCL:BOOL=!USE_NCCL! ^ "%~dp0\.." if ERRORLEVEL 1 ( From 7c1daec397855aeecf56f42cd3f7f470a4461dd5 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Wed, 15 Feb 2017 12:17:21 -0500 Subject: [PATCH 38/51] Added *.lib/*.dll/*.pyd/*.exe to .gitignore --- .gitignore | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/.gitignore b/.gitignore index ccb55594f07..2ead5c521fc 100644 --- a/.gitignore +++ b/.gitignore @@ -5,15 +5,21 @@ *.lo *.o *.cuo +*.obj # Compiled Dynamic libraries *.so *.dylib +*.dll # Compiled Static libraries *.lai *.la *.a +*.lib + +# Compiled Executables +*.exe # Compiled protocol buffers *.pb.h @@ -22,6 +28,7 @@ # Compiled python *.pyc +*.pyd # Compiled MATLAB *.mex* From ebc54c0b1ed7d68b4c6e6071196373580a33537d Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Wed, 15 Feb 2017 16:26:04 -0500 Subject: [PATCH 39/51] Added powershell scripts to mimic the .sh script to download and create the mnist database --- data/mnist/get_mnist.ps1 | 24 ++++++++++++++++++++++++ examples/mnist/create_mnist.ps1 | 33 +++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+) create mode 100644 data/mnist/get_mnist.ps1 create mode 100644 examples/mnist/create_mnist.ps1 diff --git a/data/mnist/get_mnist.ps1 b/data/mnist/get_mnist.ps1 new file mode 100644 index 00000000000..1fe464b039a --- /dev/null +++ b/data/mnist/get_mnist.ps1 @@ -0,0 +1,24 @@ +# This scripts downloads the mnist data and unzips it. +$ErrorActionPreference = 'Stop' + +pushd $PSScriptRoot + +echo "Downloading..." + +# get the path to 7-zip from the registry +$7zip = Join-Path (get-item HKLM:\SOFTWARE\7-Zip).GetValue('Path') '7z.exe' + +$fnames = @("train-images-idx3-ubyte", + "train-labels-idx1-ubyte", + "t10k-images-idx3-ubyte", + "t10k-labels-idx1-ubyte") + +foreach($fname in $fnames) { + if(-not (Test-Path $fname)) { + # Start-BitsTransfer -Source "http://yann.lecun.com/exdb/mnist/$fname.gz" -Destination "$fname.gz" + wget -Uri "http://yann.lecun.com/exdb/mnist/$fname.gz" -OutFile "$fname.gz" + . $7zip x "$fname.gz" + } +} + +popd \ No newline at end of file diff --git a/examples/mnist/create_mnist.ps1 b/examples/mnist/create_mnist.ps1 new file mode 100644 index 00000000000..8c6d72394e2 --- /dev/null +++ b/examples/mnist/create_mnist.ps1 @@ -0,0 +1,33 @@ +# This script converts the mnist data into lmdb/leveldb format, +# depending on the value assigned to $BACKEND. +param( + [string]$BuildDir +) + +$ErrorActionPreference = 'Stop' + +$CaffeRoot = Resolve-Path (Join-Path $PSScriptRoot ..\..) +$EXAMPLE = "$CaffeRoot\examples\mnist" +$DATA = "$CaffeRoot\data\mnist" +if($BuildDir -eq $null) { + $BuildDir = "$CaffeRoot\build" +} +$BUILD = "$BuildDir\examples\mnist" + +$BACKEND = "lmdb" + +echo "Creating $BACKEND..." + +if(Test-Path $EXAMPLE\mnist_train_$BACKEND) { + rm -Recurse -Force $EXAMPLE\mnist_train_$BACKEND +} +if(Test-Path $EXAMPLE\mnist_train_$BACKEND) { + rm -Recurse -Force $EXAMPLE\mnist_test_$BACKEND +} + +. $BUILD\convert_mnist_data.exe $DATA\train-images.idx3-ubyte ` + $DATA\train-labels.idx1-ubyte $EXAMPLE\mnist_train_$BACKEND --backend=$BACKEND +. $BUILD\convert_mnist_data.exe $DATA\t10k-images.idx3-ubyte ` + $DATA\t10k-labels.idx1-ubyte $EXAMPLE\mnist_test_$BACKEND --backend=$BACKEND + +echo "Done." From 8024d67924b2c22c64fcf24c0c0a4a15f9f35997 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Wed, 15 Feb 2017 16:29:50 -0500 Subject: [PATCH 40/51] Added copy of required dependencies to examples executables --- examples/CMakeLists.txt | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/examples/CMakeLists.txt b/examples/CMakeLists.txt index a59e0df36b0..3e11d6b2c6a 100644 --- a/examples/CMakeLists.txt +++ b/examples/CMakeLists.txt @@ -3,15 +3,19 @@ file(GLOB_RECURSE examples_srcs "${PROJECT_SOURCE_DIR}/examples/*.cpp") foreach(source_file ${examples_srcs}) # get file name get_filename_component(name ${source_file} NAME_WE) - + # get folder name get_filename_component(path ${source_file} PATH) get_filename_component(folder ${path} NAME_WE) - + add_executable(${name} ${source_file}) target_link_libraries(${name} ${Caffe_LINK}) caffe_default_properties(${name}) + if(MSVC AND COPY_PREREQUISITES) + caffe_copy_prerequisites(${name} USE_HARD_LINKS) + endif() + # set back RUNTIME_OUTPUT_DIRECTORY set_target_properties(${name} PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${PROJECT_BINARY_DIR}/examples/${folder}") @@ -21,6 +25,10 @@ foreach(source_file ${examples_srcs}) # install install(TARGETS ${name} DESTINATION bin) + if(MSVC AND INSTALL_PREREQUISITES) + caffe_install_prerequisites(${name} DESTINATION bin) + endif() + if(UNIX OR APPLE) # Funny command to make tutorials work # TODO: remove in future as soon as naming is standardized everywhere From 527f97c0692f116ada7cb97eed8172ef7da05416 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Fri, 3 Mar 2017 23:30:25 -0500 Subject: [PATCH 41/51] Updated prebuilt dependencies. Fixes #5348 --- cmake/WindowsDownloadPrebuiltDependencies.cmake | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/cmake/WindowsDownloadPrebuiltDependencies.cmake b/cmake/WindowsDownloadPrebuiltDependencies.cmake index 3fe2fabaf93..9e7a0166911 100644 --- a/cmake/WindowsDownloadPrebuiltDependencies.cmake +++ b/cmake/WindowsDownloadPrebuiltDependencies.cmake @@ -1,9 +1,9 @@ -set(DEPENDENCIES_URL_1800_27 "https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v120_x64_py27_1.0.1.tar.bz2") -set(DEPENDENCIES_SHA_1800_27 "3f45fe3f27b27a7809f9de1bd85e56888b01dbe2") -set(DEPENDENCIES_URL_1900_27 "https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v140_x64_py27_1.0.1.tar.bz2") -set(DEPENDENCIES_SHA_1900_27 "427faf33745cf8cd70c7d043c85db7dda7243122") -set(DEPENDENCIES_URL_1900_35 "https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v140_x64_py35_1.0.1.tar.bz2") -set(DEPENDENCIES_SHA_1900_35 "1f55dac54aeab7ae3a1cda145ca272dea606bdf9") +set(DEPENDENCIES_URL_1800_27 "https://github.com/willyd/caffe-builder/releases/download/v1.1.0/libraries_v120_x64_py27_1.1.0.tar.bz2") +set(DEPENDENCIES_SHA_1800_27 "ba833d86d19b162a04d68b09b06df5e0dad947d4") +set(DEPENDENCIES_URL_1900_27 "https://github.com/willyd/caffe-builder/releases/download/v1.1.0/libraries_v140_x64_py27_1.1.0.tar.bz2") +set(DEPENDENCIES_SHA_1900_27 "17eecb095bd3b0774a87a38624a77ce35e497cd2") +set(DEPENDENCIES_URL_1900_35 "https://github.com/willyd/caffe-builder/releases/download/v1.1.0/libraries_v140_x64_py35_1.1.0.tar.bz2") +set(DEPENDENCIES_SHA_1900_35 "f060403fd1a7448d866d27c0e5b7dced39c0a607") caffe_option(USE_PREBUILT_DEPENDENCIES "Download and use the prebuilt dependencies" ON IF MSVC) if(MSVC) From a35cc49d4b05a03e48f4b4a3f4d6becd1c1e54c7 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Sat, 11 Mar 2017 19:39:00 -0500 Subject: [PATCH 42/51] Added train_lenet.ps1, fixed create_mnist.ps1 and update mnist readme.md for windows. --- examples/mnist/create_mnist.ps1 | 4 ++-- examples/mnist/readme.md | 2 +- examples/mnist/train_lenet.ps1 | 10 ++++++++++ 3 files changed, 13 insertions(+), 3 deletions(-) create mode 100644 examples/mnist/train_lenet.ps1 diff --git a/examples/mnist/create_mnist.ps1 b/examples/mnist/create_mnist.ps1 index 8c6d72394e2..6bb4b2812ac 100644 --- a/examples/mnist/create_mnist.ps1 +++ b/examples/mnist/create_mnist.ps1 @@ -6,10 +6,10 @@ param( $ErrorActionPreference = 'Stop' -$CaffeRoot = Resolve-Path (Join-Path $PSScriptRoot ..\..) +$CaffeRoot = (Resolve-Path (Join-Path $PSScriptRoot ..\..)) $EXAMPLE = "$CaffeRoot\examples\mnist" $DATA = "$CaffeRoot\data\mnist" -if($BuildDir -eq $null) { +if("$BuildDir" -eq "") { $BuildDir = "$CaffeRoot\build" } $BUILD = "$BuildDir\examples\mnist" diff --git a/examples/mnist/readme.md b/examples/mnist/readme.md index 35952155a30..11dbdf77c84 100644 --- a/examples/mnist/readme.md +++ b/examples/mnist/readme.md @@ -8,7 +8,7 @@ priority: 1 # Training LeNet on MNIST with Caffe -We will assume that you have Caffe successfully compiled. If not, please refer to the [Installation page](/installation.html). In this tutorial, we will assume that your Caffe installation is located at `CAFFE_ROOT`. +We will assume that you have Caffe successfully compiled. If not, please refer to the [Installation page](/installation.html). In this tutorial, we will assume that your Caffe installation is located at `CAFFE_ROOT`. On Windows use the powershell (`.ps1`) instead of the bash (`.sh`) scripts. ## Prepare Datasets diff --git a/examples/mnist/train_lenet.ps1 b/examples/mnist/train_lenet.ps1 new file mode 100644 index 00000000000..912c62cea15 --- /dev/null +++ b/examples/mnist/train_lenet.ps1 @@ -0,0 +1,10 @@ +param( + [string]$BuildDir +) + +$CaffeRoot = (Resolve-Path (Join-Path $PSScriptRoot ..\..)) +if("$BuildDir" -eq "") { + $BuildDir = "$CaffeRoot\build" +} + +. $BuildDir\tools\caffe.exe train --solver=examples\mnist\lenet_solver.prototxt $args From ca360a148f70a254a7246490eccc85905e75afa0 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Thu, 16 Mar 2017 23:04:28 -0400 Subject: [PATCH 43/51] Fixed lint issue when .pb.h and .pb.cc files do not exist --- cmake/lint.cmake | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/cmake/lint.cmake b/cmake/lint.cmake index 6f86937f7ed..8cca27d5248 100644 --- a/cmake/lint.cmake +++ b/cmake/lint.cmake @@ -26,7 +26,9 @@ foreach(ext ${EXCLUDE_FILE_EXTENSTIONS}) endforeach() # exclude generated pb files -list(REMOVE_ITEM LINT_SOURCES ${EXCLUDED_FILES}) +if(EXCLUDED_FILES) + list(REMOVE_ITEM LINT_SOURCES ${EXCLUDED_FILES}) +endif() execute_process( COMMAND ${LINT_COMMAND} ${LINT_SOURCES} From de0ee29c8279cc51fa2e120f6de9e26f99e999ac Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Wed, 22 Mar 2017 08:27:15 -0400 Subject: [PATCH 44/51] Fixed wrong VS 2013 Update 5 version string. Fixes #5430 --- cmake/Dependencies.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/cmake/Dependencies.cmake b/cmake/Dependencies.cmake index dacc0b14a4e..2ea36628d42 100644 --- a/cmake/Dependencies.cmake +++ b/cmake/Dependencies.cmake @@ -10,7 +10,7 @@ list(APPEND Caffe_INCLUDE_DIRS PUBLIC ${Boost_INCLUDE_DIRS}) list(APPEND Caffe_DEFINITIONS PUBLIC -DBOOST_ALL_NO_LIB) list(APPEND Caffe_LINKER_LIBS PUBLIC ${Boost_LIBRARIES}) -if(DEFINED MSVC AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 12.0.40629.00) +if(DEFINED MSVC AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 18.0.40629.0) # Required for VS 2013 Update 4 or earlier. list(APPEND Caffe_DEFINITIONS PUBLIC -DBOOST_NO_CXX11_TEMPLATE_ALIASES) endif() From 621c5c7f06d1a3550cc64e8d0554b0e8afc71c78 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Mon, 27 Mar 2017 08:39:51 -0400 Subject: [PATCH 45/51] Removed deprecated VS based build. --- README.md | 4 - windows/Caffe.sln | 140 ---- windows/CommonSettings.props.example | 112 --- windows/CommonSettings.targets | 11 - windows/README.md | 54 -- windows/caffe/caffe.vcxproj | 121 --- windows/caffe/packages.config | 18 - windows/classification/classification.vcxproj | 112 --- windows/classification/packages.config | 18 - .../compute_image_mean/compute_image_mean.vcxproj | 112 --- windows/compute_image_mean/packages.config | 18 - .../convert_cifar_data/convert_cifar_data.vcxproj | 112 --- windows/convert_cifar_data/packages.config | 18 - windows/convert_imageset/convert_imageset.vcxproj | 112 --- windows/convert_imageset/packages.config | 18 - .../convert_mnist_data/convert_mnist_data.vcxproj | 112 --- windows/convert_mnist_data/packages.config | 18 - .../convert_mnist_siamese_data.vcxproj | 112 --- windows/convert_mnist_siamese_data/packages.config | 18 - windows/extract_features/extract_features.vcxproj | 118 --- windows/extract_features/packages.config | 18 - windows/libcaffe/libcaffe.vcxproj | 392 ---------- windows/libcaffe/libcaffe.vcxproj.filters | 821 --------------------- windows/libcaffe/packages.config | 14 - windows/matcaffe/matcaffe.def | 2 - windows/matcaffe/matcaffe.vcxproj | 128 ---- windows/matcaffe/packages.config | 18 - windows/nuget.config | 7 - windows/pycaffe/packages.config | 19 - windows/pycaffe/pycaffe.vcxproj | 129 ---- windows/scripts/BinplaceCudaDependencies.cmd | 27 - windows/scripts/FixGFlagsNaming.cmd | 24 - windows/scripts/MatlabPostBuild.cmd | 9 - windows/scripts/MatlabPreBuild.cmd | 8 - windows/scripts/ProtoCompile.cmd | 27 - windows/scripts/PythonPostBuild.cmd | 9 - windows/scripts/PythonPreBuild.cmd | 15 - windows/test_all/packages.config | 18 - windows/test_all/test_all.vcxproj | 208 ------ windows/test_all/test_all.vcxproj.filters | 235 ------ windows/upgrade_net_proto_binary/packages.config | 18 - .../upgrade_net_proto_binary.vcxproj | 112 --- windows/upgrade_net_proto_text/packages.config | 18 - .../upgrade_net_proto_text.vcxproj | 112 --- windows/upgrade_solver_proto_text/packages.config | 18 - .../upgrade_solver_proto_text.vcxproj | 112 --- 46 files changed, 3876 deletions(-) delete mode 100644 windows/Caffe.sln delete mode 100644 windows/CommonSettings.props.example delete mode 100644 windows/CommonSettings.targets delete mode 100644 windows/README.md delete mode 100644 windows/caffe/caffe.vcxproj delete mode 100644 windows/caffe/packages.config delete mode 100644 windows/classification/classification.vcxproj delete mode 100644 windows/classification/packages.config delete mode 100644 windows/compute_image_mean/compute_image_mean.vcxproj delete mode 100644 windows/compute_image_mean/packages.config delete mode 100644 windows/convert_cifar_data/convert_cifar_data.vcxproj delete mode 100644 windows/convert_cifar_data/packages.config delete mode 100644 windows/convert_imageset/convert_imageset.vcxproj delete mode 100644 windows/convert_imageset/packages.config delete mode 100644 windows/convert_mnist_data/convert_mnist_data.vcxproj delete mode 100644 windows/convert_mnist_data/packages.config delete mode 100644 windows/convert_mnist_siamese_data/convert_mnist_siamese_data.vcxproj delete mode 100644 windows/convert_mnist_siamese_data/packages.config delete mode 100644 windows/extract_features/extract_features.vcxproj delete mode 100644 windows/extract_features/packages.config delete mode 100644 windows/libcaffe/libcaffe.vcxproj delete mode 100644 windows/libcaffe/libcaffe.vcxproj.filters delete mode 100644 windows/libcaffe/packages.config delete mode 100644 windows/matcaffe/matcaffe.def delete mode 100644 windows/matcaffe/matcaffe.vcxproj delete mode 100644 windows/matcaffe/packages.config delete mode 100644 windows/nuget.config delete mode 100644 windows/pycaffe/packages.config delete mode 100644 windows/pycaffe/pycaffe.vcxproj delete mode 100644 windows/scripts/BinplaceCudaDependencies.cmd delete mode 100644 windows/scripts/FixGFlagsNaming.cmd delete mode 100644 windows/scripts/MatlabPostBuild.cmd delete mode 100644 windows/scripts/MatlabPreBuild.cmd delete mode 100644 windows/scripts/ProtoCompile.cmd delete mode 100644 windows/scripts/PythonPostBuild.cmd delete mode 100644 windows/scripts/PythonPreBuild.cmd delete mode 100644 windows/test_all/packages.config delete mode 100644 windows/test_all/test_all.vcxproj delete mode 100644 windows/test_all/test_all.vcxproj.filters delete mode 100644 windows/upgrade_net_proto_binary/packages.config delete mode 100644 windows/upgrade_net_proto_binary/upgrade_net_proto_binary.vcxproj delete mode 100644 windows/upgrade_net_proto_text/packages.config delete mode 100644 windows/upgrade_net_proto_text/upgrade_net_proto_text.vcxproj delete mode 100644 windows/upgrade_solver_proto_text/packages.config delete mode 100644 windows/upgrade_solver_proto_text/upgrade_solver_proto_text.vcxproj diff --git a/README.md b/README.md index 26b779a347a..37311bdcc65 100644 --- a/README.md +++ b/README.md @@ -117,10 +117,6 @@ CMake can be used to build a shared library instead of the default static librar Should you encounter any error please post the output of the above commands by redirecting the output to a file and open a topic on the [caffe-users list](https://groups.google.com/forum/#!forum/caffe-users) mailing list. -## Previous Visual Studio based build - -The previous windows build based on Visual Studio project files is now deprecated. However, it is still available in the `windows` folder. Please see the [README.md](windows/README.md) in there for details. - ## Known issues - The `GPUTimer` related test cases always fail on Windows. This seems to be a difference between UNIX and Windows. diff --git a/windows/Caffe.sln b/windows/Caffe.sln deleted file mode 100644 index 3a3b09d41d7..00000000000 --- a/windows/Caffe.sln +++ /dev/null @@ -1,140 +0,0 @@ - -Microsoft Visual Studio Solution File, Format Version 12.00 -# Visual Studio 2013 -VisualStudioVersion = 12.0.40629.0 -MinimumVisualStudioVersion = 10.0.40219.1 -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "libcaffe", "libcaffe\libcaffe.vcxproj", "{A9ACEF83-7B63-4574-A554-89CE869EA141}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "caffe", "caffe\caffe.vcxproj", "{CE6BBC46-9EFC-4029-9065-85A023866AFB}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "compute_image_mean", "compute_image_mean\compute_image_mean.vcxproj", "{09A8EDAC-20B9-414F-9654-961388FD5A8C}" - ProjectSection(ProjectDependencies) = postProject - {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "convert_imageset", "convert_imageset\convert_imageset.vcxproj", "{44AAEF8E-2DF2-4534-AD6C-50017997897B}" - ProjectSection(ProjectDependencies) = postProject - {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "extract_features", "extract_features\extract_features.vcxproj", "{C4A4173A-1BBA-4668-B506-0538A7D259E4}" - ProjectSection(ProjectDependencies) = postProject - {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "test_all", "test_all\test_all.vcxproj", "{00BBA8C0-707D-42A7-82FF-D5211185ED7F}" - ProjectSection(ProjectDependencies) = postProject - {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pycaffe", "pycaffe\pycaffe.vcxproj", "{38B6CE09-4B1A-4E72-A547-8A3299D8DA60}" - ProjectSection(ProjectDependencies) = postProject - {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "matcaffe", "matcaffe\matcaffe.vcxproj", "{7173D611-3A7A-4F07-943A-727C6862E8D5}" - ProjectSection(ProjectDependencies) = postProject - {CE6BBC46-9EFC-4029-9065-85A023866AFB} = {CE6BBC46-9EFC-4029-9065-85A023866AFB} - EndProjectSection -EndProject -Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "props", "props", "{632DD6E1-28DF-42F9-AD7F-1C1F2D38765C}" - ProjectSection(SolutionItems) = preProject - CommonSettings.props = CommonSettings.props - EndProjectSection -EndProject -Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "scripts", "scripts", "{E2EF4AB6-AB52-4777-9783-4669A0D61F80}" - ProjectSection(SolutionItems) = preProject - scripts\BinplaceCudaDependencies.cmd = scripts\BinplaceCudaDependencies.cmd - scripts\FixGFlagsNaming.cmd = scripts\FixGFlagsNaming.cmd - scripts\ProtoCompile.cmd = scripts\ProtoCompile.cmd - scripts\PythonPostBuild.cmd = scripts\PythonPostBuild.cmd - scripts\PythonPreBuild.cmd = scripts\PythonPreBuild.cmd - scripts\MatlabPostBuild.cmd = scripts\MatlabPostBuild.cmd - scripts\MatlabPreBuild.cmd = scripts\MatlabPreBuild.cmd - EndProjectSection -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "convert_cifar_data", "convert_cifar_data\convert_cifar_data.vcxproj", "{B166B643-C90B-4903-B735-D2D4ED4F2248}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "classification", "classification\classification.vcxproj", "{273E7766-61AA-437C-BCA9-4CA7FE0484D4}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "convert_mnist_data", "convert_mnist_data\convert_mnist_data.vcxproj", "{73EED2A0-EED0-4514-8C95-ADA25CD3C72D}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "convert_mnist_siamese_data", "convert_mnist_siamese_data\convert_mnist_siamese_data.vcxproj", "{3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "upgrade_net_proto_binary", "upgrade_net_proto_binary\upgrade_net_proto_binary.vcxproj", "{7971DD9E-FEA9-446B-B432-F3910B8B84A8}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "upgrade_net_proto_text", "upgrade_net_proto_text\upgrade_net_proto_text.vcxproj", "{4E201A07-4464-4ECF-8D5E-6B7E3B2D896B}" -EndProject -Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "upgrade_solver_proto_text", "upgrade_solver_proto_text\upgrade_solver_proto_text.vcxproj", "{E1185C4E-1AEA-4E0E-BE85-2671E065016A}" -EndProject -Global - GlobalSection(SolutionConfigurationPlatforms) = preSolution - Debug|x64 = Debug|x64 - Release|x64 = Release|x64 - EndGlobalSection - GlobalSection(ProjectConfigurationPlatforms) = postSolution - {A9ACEF83-7B63-4574-A554-89CE869EA141}.Debug|x64.ActiveCfg = Debug|x64 - {A9ACEF83-7B63-4574-A554-89CE869EA141}.Debug|x64.Build.0 = Debug|x64 - {A9ACEF83-7B63-4574-A554-89CE869EA141}.Release|x64.ActiveCfg = Release|x64 - {A9ACEF83-7B63-4574-A554-89CE869EA141}.Release|x64.Build.0 = Release|x64 - {CE6BBC46-9EFC-4029-9065-85A023866AFB}.Debug|x64.ActiveCfg = Debug|x64 - {CE6BBC46-9EFC-4029-9065-85A023866AFB}.Debug|x64.Build.0 = Debug|x64 - {CE6BBC46-9EFC-4029-9065-85A023866AFB}.Release|x64.ActiveCfg = Release|x64 - {CE6BBC46-9EFC-4029-9065-85A023866AFB}.Release|x64.Build.0 = Release|x64 - {09A8EDAC-20B9-414F-9654-961388FD5A8C}.Debug|x64.ActiveCfg = Debug|x64 - {09A8EDAC-20B9-414F-9654-961388FD5A8C}.Debug|x64.Build.0 = Debug|x64 - {09A8EDAC-20B9-414F-9654-961388FD5A8C}.Release|x64.ActiveCfg = Release|x64 - {09A8EDAC-20B9-414F-9654-961388FD5A8C}.Release|x64.Build.0 = Release|x64 - {44AAEF8E-2DF2-4534-AD6C-50017997897B}.Debug|x64.ActiveCfg = Debug|x64 - {44AAEF8E-2DF2-4534-AD6C-50017997897B}.Debug|x64.Build.0 = Debug|x64 - {44AAEF8E-2DF2-4534-AD6C-50017997897B}.Release|x64.ActiveCfg = Release|x64 - {44AAEF8E-2DF2-4534-AD6C-50017997897B}.Release|x64.Build.0 = Release|x64 - {C4A4173A-1BBA-4668-B506-0538A7D259E4}.Debug|x64.ActiveCfg = Debug|x64 - {C4A4173A-1BBA-4668-B506-0538A7D259E4}.Debug|x64.Build.0 = Debug|x64 - {C4A4173A-1BBA-4668-B506-0538A7D259E4}.Release|x64.ActiveCfg = Release|x64 - {C4A4173A-1BBA-4668-B506-0538A7D259E4}.Release|x64.Build.0 = Release|x64 - {00BBA8C0-707D-42A7-82FF-D5211185ED7F}.Debug|x64.ActiveCfg = Debug|x64 - {00BBA8C0-707D-42A7-82FF-D5211185ED7F}.Debug|x64.Build.0 = Debug|x64 - {00BBA8C0-707D-42A7-82FF-D5211185ED7F}.Release|x64.ActiveCfg = Release|x64 - {00BBA8C0-707D-42A7-82FF-D5211185ED7F}.Release|x64.Build.0 = Release|x64 - {38B6CE09-4B1A-4E72-A547-8A3299D8DA60}.Debug|x64.ActiveCfg = Debug|x64 - {38B6CE09-4B1A-4E72-A547-8A3299D8DA60}.Debug|x64.Build.0 = Debug|x64 - {38B6CE09-4B1A-4E72-A547-8A3299D8DA60}.Release|x64.ActiveCfg = Release|x64 - {38B6CE09-4B1A-4E72-A547-8A3299D8DA60}.Release|x64.Build.0 = Release|x64 - {7173D611-3A7A-4F07-943A-727C6862E8D5}.Debug|x64.ActiveCfg = Debug|x64 - {7173D611-3A7A-4F07-943A-727C6862E8D5}.Debug|x64.Build.0 = Debug|x64 - {7173D611-3A7A-4F07-943A-727C6862E8D5}.Release|x64.ActiveCfg = Release|x64 - {7173D611-3A7A-4F07-943A-727C6862E8D5}.Release|x64.Build.0 = Release|x64 - {B166B643-C90B-4903-B735-D2D4ED4F2248}.Debug|x64.ActiveCfg = Debug|x64 - {B166B643-C90B-4903-B735-D2D4ED4F2248}.Debug|x64.Build.0 = Debug|x64 - {B166B643-C90B-4903-B735-D2D4ED4F2248}.Release|x64.ActiveCfg = Release|x64 - {B166B643-C90B-4903-B735-D2D4ED4F2248}.Release|x64.Build.0 = Release|x64 - {273E7766-61AA-437C-BCA9-4CA7FE0484D4}.Debug|x64.ActiveCfg = Debug|x64 - {273E7766-61AA-437C-BCA9-4CA7FE0484D4}.Debug|x64.Build.0 = Debug|x64 - {273E7766-61AA-437C-BCA9-4CA7FE0484D4}.Release|x64.ActiveCfg = Release|x64 - {273E7766-61AA-437C-BCA9-4CA7FE0484D4}.Release|x64.Build.0 = Release|x64 - {73EED2A0-EED0-4514-8C95-ADA25CD3C72D}.Debug|x64.ActiveCfg = Debug|x64 - {73EED2A0-EED0-4514-8C95-ADA25CD3C72D}.Debug|x64.Build.0 = Debug|x64 - {73EED2A0-EED0-4514-8C95-ADA25CD3C72D}.Release|x64.ActiveCfg = Release|x64 - {73EED2A0-EED0-4514-8C95-ADA25CD3C72D}.Release|x64.Build.0 = Release|x64 - {3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D}.Debug|x64.ActiveCfg = Debug|x64 - {3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D}.Debug|x64.Build.0 = Debug|x64 - {3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D}.Release|x64.ActiveCfg = Release|x64 - {3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D}.Release|x64.Build.0 = Release|x64 - {7971DD9E-FEA9-446B-B432-F3910B8B84A8}.Debug|x64.ActiveCfg = Debug|x64 - {7971DD9E-FEA9-446B-B432-F3910B8B84A8}.Debug|x64.Build.0 = Debug|x64 - {7971DD9E-FEA9-446B-B432-F3910B8B84A8}.Release|x64.ActiveCfg = Release|x64 - {7971DD9E-FEA9-446B-B432-F3910B8B84A8}.Release|x64.Build.0 = Release|x64 - {4E201A07-4464-4ECF-8D5E-6B7E3B2D896B}.Debug|x64.ActiveCfg = Debug|x64 - {4E201A07-4464-4ECF-8D5E-6B7E3B2D896B}.Debug|x64.Build.0 = Debug|x64 - {4E201A07-4464-4ECF-8D5E-6B7E3B2D896B}.Release|x64.ActiveCfg = Release|x64 - {4E201A07-4464-4ECF-8D5E-6B7E3B2D896B}.Release|x64.Build.0 = Release|x64 - {E1185C4E-1AEA-4E0E-BE85-2671E065016A}.Debug|x64.ActiveCfg = Debug|x64 - {E1185C4E-1AEA-4E0E-BE85-2671E065016A}.Debug|x64.Build.0 = Debug|x64 - {E1185C4E-1AEA-4E0E-BE85-2671E065016A}.Release|x64.ActiveCfg = Release|x64 - {E1185C4E-1AEA-4E0E-BE85-2671E065016A}.Release|x64.Build.0 = Release|x64 - EndGlobalSection - GlobalSection(SolutionProperties) = preSolution - HideSolutionNode = FALSE - EndGlobalSection -EndGlobal diff --git a/windows/CommonSettings.props.example b/windows/CommonSettings.props.example deleted file mode 100644 index ceb9949ea1f..00000000000 --- a/windows/CommonSettings.props.example +++ /dev/null @@ -1,112 +0,0 @@ - - - - - $(SolutionDir)..\Build - - false - true - 7.5 - - false - - false - - - - compute_35,sm_35;compute_52,sm_52 - - - - $(SolutionDir)\scripts - - - cublas.lib;cuda.lib;curand.lib;cudart.lib - - - - cudnn.lib;$(CudaDependencies) - - - $(CuDnnPath)\cuda\lib\x64;$(LibraryPath) - $(CuDnnPath)\cuda\include;$(IncludePath) - - - - $(BuildDir)\$(Platform)\$(Configuration)\ - $(BuildDir)\Int\$(ProjectName)\$(Platform)\$(Configuration)\ - - - $(OutDir);$(CUDA_PATH)\lib\$(Platform);$(LibraryPath) - $(SolutionDir)..\include;$(SolutionDir)..\include\caffe\proto;$(CUDA_PATH)\include;$(IncludePath) - - - C:\Miniconda2\ - $(PythonDir)\libs;$(LibraryPath) - $(PythonDir)\include;$(IncludePath) - - - C:\Program Files\MATLAB\R2014b - $(MatlabDir)\extern\lib\win64\microsoft;$(LibraryPath) - $(MatlabDir)\extern\include;$(IncludePath) - - - - CPU_ONLY;%(PreprocessorDefinitions) - - - - - USE_CUDNN;%(PreprocessorDefinitions) - - - USE_CUDNN - - - - - WITH_PYTHON_LAYER;BOOST_PYTHON_STATIC_LIB;%(PreprocessorDefinitions) - - - - - MATLAB_MEX_FILE;%(PreprocessorDefinitions) - - - - - false - true - _SCL_SECURE_NO_WARNINGS;USE_OPENCV;USE_LEVELDB;USE_LMDB;%(PreprocessorDefinitions) - true - - - - - Full - NDEBUG;%(PreprocessorDefinitions) - MultiThreadedDLL - true - - - true - true - UseLinkTimeCodeGeneration - true - - - - - Disabled - _DEBUG;%(PreprocessorDefinitions) - MultiThreadedDebugDLL - - - true - - - diff --git a/windows/CommonSettings.targets b/windows/CommonSettings.targets deleted file mode 100644 index b9077d354b7..00000000000 --- a/windows/CommonSettings.targets +++ /dev/null @@ -1,11 +0,0 @@ - - - - - - - - - \ No newline at end of file diff --git a/windows/README.md b/windows/README.md deleted file mode 100644 index 6b94121c940..00000000000 --- a/windows/README.md +++ /dev/null @@ -1,54 +0,0 @@ -# Windows Caffe - -This is the old Visual Studio based build of caffe. The procedure below was left here for reference and may not work. This build will be removed in the near future in favor of the CMake based build. - -## Windows Setup -**Requirements**: Visual Studio 2013 - -### Pre-Build Steps -Copy `.\windows\CommonSettings.props.example` to `.\windows\CommonSettings.props` - -By defaults Windows build requires `CUDA` and `cuDNN` libraries. -Both can be disabled by adjusting build variables in `.\windows\CommonSettings.props`. -Python support is disabled by default, but can be enabled via `.\windows\CommonSettings.props` as well. -3rd party dependencies required by Caffe are automatically resolved via NuGet. - -### CUDA -Download `CUDA Toolkit 7.5` [from nVidia website](https://developer.nvidia.com/cuda-toolkit). -If you don't have CUDA installed, you can experiment with CPU_ONLY build. -In `.\windows\CommonSettings.props` set `CpuOnlyBuild` to `true` and set `UseCuDNN` to `false`. - -### cuDNN -Download `cuDNN v3` or `cuDNN v4` [from nVidia website](https://developer.nvidia.com/cudnn). -Unpack downloaded zip to %CUDA_PATH% (environment variable set by CUDA installer). -Alternatively, you can unpack zip to any location and set `CuDnnPath` to point to this location in `.\windows\CommonSettings.props`. -`CuDnnPath` defined in `.\windows\CommonSettings.props`. -Also, you can disable cuDNN by setting `UseCuDNN` to `false` in the property file. - -### Python -To build Caffe Python wrapper set `PythonSupport` to `true` in `.\windows\CommonSettings.props`. -Download Miniconda 2.7 64-bit Windows installer [from Miniconda website] (http://conda.pydata.org/miniconda.html). -Install for all users and add Python to PATH (through installer). - -Run the following commands from elevated command prompt: - -``` -conda install --yes numpy scipy matplotlib scikit-image pip -pip install protobuf -``` - -#### Remark -After you have built solution with Python support, in order to use it you have to either: -* set `PythonPath` environment variable to point to `\Build\x64\Release\pycaffe`, or -* copy folder `\Build\x64\Release\pycaffe\caffe` under `\lib\site-packages`. - -### Matlab -To build Caffe Matlab wrapper set `MatlabSupport` to `true` and `MatlabDir` to the root of your Matlab installation in `.\windows\CommonSettings.props`. - -#### Remark -After you have built solution with Matlab support, in order to use it you have to: -* add the generated `matcaffe` folder to Matlab search path, and -* add `\Build\x64\Release` to your system path. - -### Build -Now, you should be able to build `.\windows\Caffe.sln` \ No newline at end of file diff --git a/windows/caffe/caffe.vcxproj b/windows/caffe/caffe.vcxproj deleted file mode 100644 index d445970cc32..00000000000 --- a/windows/caffe/caffe.vcxproj +++ /dev/null @@ -1,121 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {CE6BBC46-9EFC-4029-9065-85A023866AFB} - Win32Proj - x64 - caffe - 82610725 - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - "$(ScriptsDir)\FixGFlagsNaming.cmd" "$(OutDir)" $(Configuration) - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - "$(ScriptsDir)\FixGFlagsNaming.cmd" "$(OutDir)" $(Configuration) - - - NDEBUG;%(PreprocessorDefinitions);CAFFE_VERSION=1.0.0-rc3 - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/caffe/packages.config b/windows/caffe/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/caffe/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/classification/classification.vcxproj b/windows/classification/classification.vcxproj deleted file mode 100644 index a607bf93a6e..00000000000 --- a/windows/classification/classification.vcxproj +++ /dev/null @@ -1,112 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {273E7766-61AA-437C-BCA9-4CA7FE0484D4} - Win32Proj - x64 - classification - f6e60ad8 - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/classification/packages.config b/windows/classification/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/classification/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/compute_image_mean/compute_image_mean.vcxproj b/windows/compute_image_mean/compute_image_mean.vcxproj deleted file mode 100644 index 776e88bfbb5..00000000000 --- a/windows/compute_image_mean/compute_image_mean.vcxproj +++ /dev/null @@ -1,112 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {09A8EDAC-20B9-414F-9654-961388FD5A8C} - Win32Proj - x64 - compute_image_mean - 9b72fdf3 - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/compute_image_mean/packages.config b/windows/compute_image_mean/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/compute_image_mean/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/convert_cifar_data/convert_cifar_data.vcxproj b/windows/convert_cifar_data/convert_cifar_data.vcxproj deleted file mode 100644 index 90fe7d70dd4..00000000000 --- a/windows/convert_cifar_data/convert_cifar_data.vcxproj +++ /dev/null @@ -1,112 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {B166B643-C90B-4903-B735-D2D4ED4F2248} - Win32Proj - x64 - convert_cifar_data - f6e60ad8 - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/convert_cifar_data/packages.config b/windows/convert_cifar_data/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/convert_cifar_data/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/convert_imageset/convert_imageset.vcxproj b/windows/convert_imageset/convert_imageset.vcxproj deleted file mode 100644 index 4e0ab62eee4..00000000000 --- a/windows/convert_imageset/convert_imageset.vcxproj +++ /dev/null @@ -1,112 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {44AAEF8E-2DF2-4534-AD6C-50017997897B} - Win32Proj - x64 - convert_imageset - 267c8bf4 - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/convert_imageset/packages.config b/windows/convert_imageset/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/convert_imageset/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/convert_mnist_data/convert_mnist_data.vcxproj b/windows/convert_mnist_data/convert_mnist_data.vcxproj deleted file mode 100644 index e58e7a767bf..00000000000 --- a/windows/convert_mnist_data/convert_mnist_data.vcxproj +++ /dev/null @@ -1,112 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {73EED2A0-EED0-4514-8C95-ADA25CD3C72D} - Win32Proj - x64 - convert_mnist_data - f6e60ad8 - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/convert_mnist_data/packages.config b/windows/convert_mnist_data/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/convert_mnist_data/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/convert_mnist_siamese_data/convert_mnist_siamese_data.vcxproj b/windows/convert_mnist_siamese_data/convert_mnist_siamese_data.vcxproj deleted file mode 100644 index d437e7d0a48..00000000000 --- a/windows/convert_mnist_siamese_data/convert_mnist_siamese_data.vcxproj +++ /dev/null @@ -1,112 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {3FC9FE87-557C-4BA3-97C1-A71E95DC3C2D} - Win32Proj - x64 - convert_mnist_siamese_data - f6e60ad8 - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/convert_mnist_siamese_data/packages.config b/windows/convert_mnist_siamese_data/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/convert_mnist_siamese_data/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/extract_features/extract_features.vcxproj b/windows/extract_features/extract_features.vcxproj deleted file mode 100644 index 7233b9b2b96..00000000000 --- a/windows/extract_features/extract_features.vcxproj +++ /dev/null @@ -1,118 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {C4A4173A-1BBA-4668-B506-0538A7D259E4} - Win32Proj - x64 - extract_features - 8be3cb47 - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - 4005 - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - 4005 - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/extract_features/packages.config b/windows/extract_features/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/extract_features/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/libcaffe/libcaffe.vcxproj b/windows/libcaffe/libcaffe.vcxproj deleted file mode 100644 index 139ccedb202..00000000000 --- a/windows/libcaffe/libcaffe.vcxproj +++ /dev/null @@ -1,392 +0,0 @@ - - - - - - - - - Debug - x64 - - - Release - x64 - - - - {A9ACEF83-7B63-4574-A554-89CE869EA141} - libcaffe - v120 - - - - StaticLibrary - true - Unicode - - - StaticLibrary - false - true - Unicode - - - - - - - 0c91d16f - - - - - - - - true - Console - - - "$(ScriptsDir)\ProtoCompile.cmd" "$(SolutionDir)" "$(ProtocDir)" - - - "$(ScriptsDir)\BinplaceCudaDependencies.cmd" "$(CudaToolkitBinDir)" "$(CuDnnPath)" $(CpuOnlyBuild) $(UseCuDNN) "$(OutDir)" - - - 64 - $(CudaArchitecture) - true - -Xcudafe "--diag_suppress=exception_spec_override_incompat --diag_suppress=useless_using_declaration --diag_suppress=field_without_dll_interface" -D_SCL_SECURE_NO_WARNINGS -DGFLAGS_DLL_DECL= - - - 4661;4005;4812;4715;%(DisableSpecificWarnings) - $(ProjectDir)\..\..\src\;%(AdditionalIncludeDirectories) - - - /ignore:4221 %(AdditionalOptions) - - - - - Console - - - "$(ScriptsDir)\ProtoCompile.cmd" "$(SolutionDir)" "$(ProtocDir)" - - - "$(ScriptsDir)\BinplaceCudaDependencies.cmd" "$(CudaToolkitBinDir)" "$(CuDnnPath)" $(CpuOnlyBuild) $(UseCuDNN) "$(OutDir)" - - - 64 - $(CudaArchitecture) - -Xcudafe "--diag_suppress=exception_spec_override_incompat --diag_suppress=useless_using_declaration --diag_suppress=field_without_dll_interface" -D_SCL_SECURE_NO_WARNINGS -DGFLAGS_DLL_DECL= - - - 4661;4005;4812;4715;%(DisableSpecificWarnings) - $(ProjectDir)\..\..\src\;%(AdditionalIncludeDirectories) - - - /ignore:4221 %(AdditionalOptions) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/libcaffe/libcaffe.vcxproj.filters b/windows/libcaffe/libcaffe.vcxproj.filters deleted file mode 100644 index 0a7244d49f5..00000000000 --- a/windows/libcaffe/libcaffe.vcxproj.filters +++ /dev/null @@ -1,821 +0,0 @@ - - - - - {253af030-e1e0-426c-9a22-6315b0d2dab7} - - - {36c36b62-e801-40f2-bba9-a79f09fa4dba} - - - {66b19093-f1ad-443e-b5d3-f55955ff0ae2} - - - {3be25bf1-cf46-47da-b1ff-30cb442da7c5} - - - {9e47fb53-4e3b-4e03-b677-a58cc26af7fb} - - - {bbb6f6f1-8a55-469b-8729-a61f87d6b63d} - - - {f9e33710-c82c-4808-90e7-96620a190b3c} - - - {9a64cba7-8bef-4df3-b933-adec019daadb} - - - {96fba2c6-dad0-4766-b354-08a7768d57d8} - - - {e4995612-1b91-40ea-9756-44382eddca40} - - - {c820c58e-d861-4d88-8b18-2180996d0657} - - - {f10cfd17-81b6-4a08-829d-1a1fa4769d2e} - - - {fcb8114c-3425-41da-b30a-af2cb33dd851} - - - - - src\util - - - src\util - - - src\util - - - src\util - - - src\util - - - src\util - - - src\proto - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src\util - - - src\util - - - src\util - - - src\util - - - src\util - - - src - - - src - - - src - - - src\util - - - src\solvers - - - src\solvers - - - src\solvers - - - src\solvers - - - src\solvers - - - src\solvers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\layers - - - src\util - - - src - - - - - include\proto - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\util - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include\layers - - - include - - - include - - - include - - - include - - - include - - - include - - - include - - - include - - - include - - - include - - - include - - - include - - - include - - - include - - - include - - - include\layers - - - include\layers - - - include\layers - - - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\util - - - cu\util - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\layers - - - cu\solvers - - - cu\solvers - - - cu\solvers - - - cu\solvers - - - cu\solvers - - - cu\solvers - - - - - - \ No newline at end of file diff --git a/windows/libcaffe/packages.config b/windows/libcaffe/packages.config deleted file mode 100644 index 3d67f16ed6c..00000000000 --- a/windows/libcaffe/packages.config +++ /dev/null @@ -1,14 +0,0 @@ - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/matcaffe/matcaffe.def b/windows/matcaffe/matcaffe.def deleted file mode 100644 index 4b20ee249fa..00000000000 --- a/windows/matcaffe/matcaffe.def +++ /dev/null @@ -1,2 +0,0 @@ -LIBRARY "caffe_.mexw64" -EXPORTS mexFunction diff --git a/windows/matcaffe/matcaffe.vcxproj b/windows/matcaffe/matcaffe.vcxproj deleted file mode 100644 index e127b10881f..00000000000 --- a/windows/matcaffe/matcaffe.vcxproj +++ /dev/null @@ -1,128 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {7173D611-3A7A-4F07-943A-727C6862E8D5} - matcaffe - - - - v120 - DynamicLibrary - - - - - - - - - .mexw64 - caffe_ - - - - libcaffe.lib;libmx.lib;libmex.lib;$(CudaDependencies);%(AdditionalDependencies) - - - - - libcaffe.lib;libmx.lib;libmex.lib;$(CudaDependencies);%(AdditionalDependencies) - - - - - 4003 - - - "$(ScriptsDir)\MatlabPreBuild.cmd" "$(SolutionDir)" "$(OutDir)" - - - "$(ScriptsDir)\MatlabPostBuild.cmd" "$(SolutionDir)" "$(OutDir)" - - - matcaffe.def - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - - $(BuildDependsOn) - OriginalBuild;SkipBuild - 5d60c5dd - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/matcaffe/packages.config b/windows/matcaffe/packages.config deleted file mode 100644 index 920090a85a5..00000000000 --- a/windows/matcaffe/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - diff --git a/windows/nuget.config b/windows/nuget.config deleted file mode 100644 index ea7ca993c5a..00000000000 --- a/windows/nuget.config +++ /dev/null @@ -1,7 +0,0 @@ - - - - - - ..\..\NugetPackages - \ No newline at end of file diff --git a/windows/pycaffe/packages.config b/windows/pycaffe/packages.config deleted file mode 100644 index e0f4af8edaa..00000000000 --- a/windows/pycaffe/packages.config +++ /dev/null @@ -1,19 +0,0 @@ - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/pycaffe/pycaffe.vcxproj b/windows/pycaffe/pycaffe.vcxproj deleted file mode 100644 index ccf45167202..00000000000 --- a/windows/pycaffe/pycaffe.vcxproj +++ /dev/null @@ -1,129 +0,0 @@ - - - - - - - - - Debug - x64 - - - Release - x64 - - - - {38B6CE09-4B1A-4E72-A547-8A3299D8DA60} - pycaffe - - - - v120 - DynamicLibrary - - - - - - - - - .pyd - _caffe - - - $(PythonDir)\Lib\site-packages\numpy\core\include\;$(IncludePath) - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - - - - - 4003 - - - "$(ScriptsDir)\PythonPreBuild.cmd" "$(SolutionDir)" "$(ProtocDir)" "$(OutDir)" - - - "$(ScriptsDir)\PythonPostBuild.cmd" "$(SolutionDir)" "$(OutDir)" - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - $(BuildDependsOn) - OriginalBuild;SkipBuild - ce4167c6 - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/scripts/BinplaceCudaDependencies.cmd b/windows/scripts/BinplaceCudaDependencies.cmd deleted file mode 100644 index d984102882c..00000000000 --- a/windows/scripts/BinplaceCudaDependencies.cmd +++ /dev/null @@ -1,27 +0,0 @@ -set CUDA_TOOLKIT_BIN_DIR=%~1% -set CUDNN_PATH=%~2% -set IS_CPU_ONLY_BUILD=%3% -set USE_CUDNN=%4% -set OUTPUT_DIR=%~5% - -if %IS_CPU_ONLY_BUILD% == true ( - echo BinplaceCudaDependencies : CPU only build, don't copy cuda dependencies. - ) else ( - echo BinplaceCudaDependencies : Copy cudart*.dll, cublas*dll, curand*.dll to output. - - copy /y "%CUDA_TOOLKIT_BIN_DIR%\cudart*.dll" "%OUTPUT_DIR%" - copy /y "%CUDA_TOOLKIT_BIN_DIR%\cublas*.dll" "%OUTPUT_DIR%" - copy /y "%CUDA_TOOLKIT_BIN_DIR%\curand*.dll" "%OUTPUT_DIR%" - - if %USE_CUDNN% == true ( - echo BinplaceCudaDependencies : Copy cudnn*.dll to output. - - if "%CUDNN_PATH%" == "" ( - copy /y "%CUDA_TOOLKIT_BIN_DIR%\cudnn*.dll" "%OUTPUT_DIR%" - ) else ( - copy /y "%CUDNN_PATH%\cuda\bin\cudnn*.dll" "%OUTPUT_DIR%" - ) - ) else ( - echo BinplaceCudaDependencies : cuDNN isn't enabled. - ) -) \ No newline at end of file diff --git a/windows/scripts/FixGFlagsNaming.cmd b/windows/scripts/FixGFlagsNaming.cmd deleted file mode 100644 index 2dc113325ab..00000000000 --- a/windows/scripts/FixGFlagsNaming.cmd +++ /dev/null @@ -1,24 +0,0 @@ -:: Glog nuget package has dependency on GFlags nuget package -:: Caffe also has direct dependency on GFlags -:: Unfortunately in GLog nuget package, dependency to GFlags dll was incorrectly set (naming is wrong) -:: For this reasons Caffe needs gflags.dll/gflagsd.dll in release/debug -:: and GLog needs libgflags.dll/libgflags-debug.dll in release/debug -:: This scripts is a workaround for this issue. - -set OUTPUT_DIR=%~1% -set BUILD_CONFIG=%2% - -if %BUILD_CONFIG% == Release ( - set originalDllName=gflags.dll - set newDllName=libgflags.dll -) else ( - set originalDllName=gflagsd.dll - set newDllName=libgflags-debug.dll -) - -if exist "%OUTPUT_DIR%\%newDllName%" ( - echo FixGFlagsNaming.cmd : "%newDllName%" already exists -) else ( - echo FixGFlagsNaming.cmd : mklink /H "%OUTPUT_DIR%\%newDllName%" "%OUTPUT_DIR%\%originalDllName%" - mklink /H "%OUTPUT_DIR%\%newDllName%" "%OUTPUT_DIR%\%originalDllName%" -) \ No newline at end of file diff --git a/windows/scripts/MatlabPostBuild.cmd b/windows/scripts/MatlabPostBuild.cmd deleted file mode 100644 index fac2874caba..00000000000 --- a/windows/scripts/MatlabPostBuild.cmd +++ /dev/null @@ -1,9 +0,0 @@ -set SOLUTION_DIR=%~1% -set OUTPUT_DIR=%~2% - -echo MatlabPostBuild.cmd : copy matlab generated scripts to output. - -@echo run_tests.m > "%temp%\excludelist.txt" -xcopy /y "%SOLUTION_DIR%..\matlab\+caffe\*.m" "%OUTPUT_DIR%matcaffe\+caffe" /exclude:%temp%\excludelist.txt -copy /y "%SOLUTION_DIR%..\matlab\+caffe\private\*.m" "%OUTPUT_DIR%matcaffe\+caffe\private" -move /y "%OUTPUT_DIR%caffe_.*" "%OUTPUT_DIR%matcaffe\+caffe\private" diff --git a/windows/scripts/MatlabPreBuild.cmd b/windows/scripts/MatlabPreBuild.cmd deleted file mode 100644 index 8d1cb5ff73b..00000000000 --- a/windows/scripts/MatlabPreBuild.cmd +++ /dev/null @@ -1,8 +0,0 @@ -set SOLUTION_DIR=%~1% -set OUTPUT_DIR=%~2% - -echo MatlabPreBuild.cmd : Create output directories for matlab scripts. - -if not exist "%OUTPUT_DIR%\matcaffe" mkdir "%OUTPUT_DIR%\matcaffe" -if not exist "%OUTPUT_DIR%\matcaffe\+caffe" mkdir "%OUTPUT_DIR%\matcaffe\+caffe" -if not exist "%OUTPUT_DIR%\matcaffe\+caffe\private" mkdir "%OUTPUT_DIR%\matcaffe\+caffe\private" diff --git a/windows/scripts/ProtoCompile.cmd b/windows/scripts/ProtoCompile.cmd deleted file mode 100644 index d056e6a17c0..00000000000 --- a/windows/scripts/ProtoCompile.cmd +++ /dev/null @@ -1,27 +0,0 @@ -set SOLUTION_DIR=%~1% -set PROTO_DIR=%~2% - -set INCLUDE_PROTO_DIR=%SOLUTION_DIR%..\include\caffe\proto -SET SRC_PROTO_DIR=%SOLUTION_DIR%..\src\caffe\proto -set PROTO_TEMP_DIR=%SRC_PROTO_DIR%\temp - -echo ProtoCompile.cmd : Create proto temp directory "%PROTO_TEMP_DIR%" -mkdir "%PROTO_TEMP_DIR%" - -echo ProtoCompile.cmd : Generating "%PROTO_TEMP_DIR%\caffe.pb.h" and "%PROTO_TEMP_DIR%\caffe.pb.cc" -"%PROTO_DIR%protoc" --proto_path="%SRC_PROTO_DIR%" --cpp_out="%PROTO_TEMP_DIR%" "%SRC_PROTO_DIR%\caffe.proto" - -echo ProtoCompile.cmd : Create proto include directory -mkdir "%INCLUDE_PROTO_DIR%" - -echo ProtoCompile.cmd : Compare newly compiled caffe.pb.h with existing one -fc /b "%PROTO_TEMP_DIR%\caffe.pb.h" "%INCLUDE_PROTO_DIR%\caffe.pb.h" > NUL - -if errorlevel 1 ( - echo ProtoCompile.cmd : Move newly generated caffe.pb.h to "%INCLUDE_PROTO_DIR%\caffe.pb.h" - echo ProtoCompile.cmd : and caffe.pb.cc to "%SRC_PROTO_DIR%\caffe.pb.cc" - move /y "%PROTO_TEMP_DIR%\caffe.pb.h" "%INCLUDE_PROTO_DIR%\caffe.pb.h" - move /y "%PROTO_TEMP_DIR%\caffe.pb.cc" "%SRC_PROTO_DIR%\caffe.pb.cc" -) - -rmdir /S /Q "%PROTO_TEMP_DIR%" \ No newline at end of file diff --git a/windows/scripts/PythonPostBuild.cmd b/windows/scripts/PythonPostBuild.cmd deleted file mode 100644 index 28ebcb844d7..00000000000 --- a/windows/scripts/PythonPostBuild.cmd +++ /dev/null @@ -1,9 +0,0 @@ -set SOLUTION_DIR=%~1% -set OUTPUT_DIR=%~2% - -echo PythonPostBuild.cmd : copy python generated scripts to output. - -copy /y "%SOLUTION_DIR%..\python\caffe\*.py" "%OUTPUT_DIR%pycaffe\caffe" -copy /y "%SOLUTION_DIR%..\python\*.py" "%OUTPUT_DIR%pycaffe" -move /y "%OUTPUT_DIR%_caffe.*" "%OUTPUT_DIR%pycaffe\caffe" -copy /y "%OUTPUT_DIR%\*.dll" "%OUTPUT_DIR%pycaffe\caffe" \ No newline at end of file diff --git a/windows/scripts/PythonPreBuild.cmd b/windows/scripts/PythonPreBuild.cmd deleted file mode 100644 index 1f07b1d2f3b..00000000000 --- a/windows/scripts/PythonPreBuild.cmd +++ /dev/null @@ -1,15 +0,0 @@ -set SOLUTION_DIR=%~1% -set PROTO_COMPILER_DIR=%~2% -set OUTPUT_DIR=%~3% - -echo PythonPreBuild.cmd : Create output directories for python scripts. - -if not exist "%OUTPUT_DIR%\pycaffe" mkdir "%OUTPUT_DIR%\pycaffe" -if not exist "%OUTPUT_DIR%\pycaffe\caffe" mkdir "%OUTPUT_DIR%\pycaffe\caffe" -if not exist "%OUTPUT_DIR%\pycaffe\caffe\proto" mkdir "%OUTPUT_DIR%\pycaffe\caffe\proto" - -echo PythonPreBuild.cmd : Create dummy __init__.py file -rem. > "%OUTPUT_DIR%\pycaffe\caffe\proto\__init__.py" - -echo PythonPreBuild.cmd : Generating src\caffe\proto\caffe.pb.h with python bindings -"%PROTO_COMPILER_DIR%\protoc" "%SOLUTION_DIR%\..\src\caffe\proto\caffe.proto" --proto_path="%SOLUTION_DIR%\..\src\caffe\proto" --python_out="%OUTPUT_DIR%\pycaffe\caffe\proto" \ No newline at end of file diff --git a/windows/test_all/packages.config b/windows/test_all/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/test_all/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/test_all/test_all.vcxproj b/windows/test_all/test_all.vcxproj deleted file mode 100644 index 7761e6b86f2..00000000000 --- a/windows/test_all/test_all.vcxproj +++ /dev/null @@ -1,208 +0,0 @@ - - - - - - - - - Debug - x64 - - - Release - x64 - - - - {00BBA8C0-707D-42A7-82FF-D5211185ED7F} - Win32Proj - x64 - test_all - 1df3590e - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - 4005;%(DisableSpecificWarnings) - $(ProjectDir)\..\..\src;%(AdditionalIncludeDirectories) - - - 64 - $(CudaArchitecture) - true - -Xcudafe "--diag_suppress=exception_spec_override_incompat --diag_suppress=useless_using_declaration --diag_suppress=field_without_dll_interface --diag_suppress=boolean_controlling_expr_is_constant" -D_SCL_SECURE_NO_WARNINGS -DGFLAGS_DLL_DECL= - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - 4005;%(DisableSpecificWarnings) - $(ProjectDir)\..\..\src;%(AdditionalIncludeDirectories) - - - 64 - $(CudaArchitecture) - -Xcudafe "--diag_suppress=exception_spec_override_incompat --diag_suppress=useless_using_declaration --diag_suppress=field_without_dll_interface --diag_suppress=boolean_controlling_expr_is_constant" -D_SCL_SECURE_NO_WARNINGS -DGFLAGS_DLL_DECL= - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Document - - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/test_all/test_all.vcxproj.filters b/windows/test_all/test_all.vcxproj.filters deleted file mode 100644 index 46811c42ed0..00000000000 --- a/windows/test_all/test_all.vcxproj.filters +++ /dev/null @@ -1,235 +0,0 @@ - - - - - {4FC737F1-C7A5-4376-A066-2A32D752A2FF} - cpp;c;cc;cxx;def;odl;idl;hpj;bat;asm;asmx - - - {93995380-89BD-4b04-88EB-625FBE52EBFB} - h;hh;hpp;hxx;hm;inl;inc;xsd - - - {46116906-a399-42c7-be9d-8a20cbbb0169} - - - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - src - - - - - include - - - include - - - - - - - - cu - - - \ No newline at end of file diff --git a/windows/upgrade_net_proto_binary/packages.config b/windows/upgrade_net_proto_binary/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/upgrade_net_proto_binary/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/upgrade_net_proto_binary/upgrade_net_proto_binary.vcxproj b/windows/upgrade_net_proto_binary/upgrade_net_proto_binary.vcxproj deleted file mode 100644 index 65f3b7e84f8..00000000000 --- a/windows/upgrade_net_proto_binary/upgrade_net_proto_binary.vcxproj +++ /dev/null @@ -1,112 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {7971DD9E-FEA9-446B-B432-F3910B8B84A8} - Win32Proj - x64 - upgrade_net_proto_binary - f6e60ad8 - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/upgrade_net_proto_text/packages.config b/windows/upgrade_net_proto_text/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/upgrade_net_proto_text/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/upgrade_net_proto_text/upgrade_net_proto_text.vcxproj b/windows/upgrade_net_proto_text/upgrade_net_proto_text.vcxproj deleted file mode 100644 index 2cd46cfc5e3..00000000000 --- a/windows/upgrade_net_proto_text/upgrade_net_proto_text.vcxproj +++ /dev/null @@ -1,112 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {4E201A07-4464-4ECF-8D5E-6B7E3B2D896B} - Win32Proj - x64 - upgrade_net_proto_text - f6e60ad8 - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/upgrade_solver_proto_text/packages.config b/windows/upgrade_solver_proto_text/packages.config deleted file mode 100644 index 25a7e34d470..00000000000 --- a/windows/upgrade_solver_proto_text/packages.config +++ /dev/null @@ -1,18 +0,0 @@ - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/windows/upgrade_solver_proto_text/upgrade_solver_proto_text.vcxproj b/windows/upgrade_solver_proto_text/upgrade_solver_proto_text.vcxproj deleted file mode 100644 index 239f2fbf802..00000000000 --- a/windows/upgrade_solver_proto_text/upgrade_solver_proto_text.vcxproj +++ /dev/null @@ -1,112 +0,0 @@ - - - - - - - - Debug - x64 - - - Release - x64 - - - - {E1185C4E-1AEA-4E0E-BE85-2671E065016A} - Win32Proj - x64 - upgrade_solver_proto_text - f6e60ad8 - - - - Application - true - Unicode - v120 - - - Application - false - Unicode - v120 - - - - - - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - libcaffe.lib;$(CudaDependencies);%(AdditionalDependencies) - Console - - - - - - - - {a9acef83-7b63-4574-a554-89ce869ea141} - false - true - false - true - true - - - - - - - - - - - - - - - - - - - - - - - - - - - This project references NuGet package(s) that are missing on this computer. Enable NuGet Package Restore to download them. For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}. - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file From d80035784c4cf0952b1ceccde6965bc0489c8fff Mon Sep 17 00:00:00 2001 From: AhmedOS Date: Tue, 28 Mar 2017 18:00:02 +0200 Subject: [PATCH 46/51] Dependencies URLs updated for newer version --- scripts/download_prebuilt_dependencies.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/scripts/download_prebuilt_dependencies.py b/scripts/download_prebuilt_dependencies.py index f3d31147ca0..01000f17575 100644 --- a/scripts/download_prebuilt_dependencies.py +++ b/scripts/download_prebuilt_dependencies.py @@ -12,12 +12,12 @@ from download_model_binary import reporthook WIN_DEPENDENCIES_URLS = { - ('v120', '2.7'):("https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v120_x64_py27_1.0.1.tar.bz2", - "3f45fe3f27b27a7809f9de1bd85e56888b01dbe2"), - ('v140', '2.7'):("https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v140_x64_py27_1.0.1.tar.bz2", - "427faf33745cf8cd70c7d043c85db7dda7243122"), - ('v140', '3.5'):("https://github.com/willyd/caffe-builder/releases/download/v1.0.1/libraries_v140_x64_py35_1.0.1.tar.bz2", - "1f55dac54aeab7ae3a1cda145ca272dea606bdf9"), + ('v120', '2.7'):("https://github.com/willyd/caffe-builder/releases/download/v1.1.0/libraries_v120_x64_py27_1.1.0.tar.bz2", + "ba833d86d19b162a04d68b09b06df5e0dad947d4"), + ('v140', '2.7'):("https://github.com/willyd/caffe-builder/releases/download/v1.1.0/libraries_v140_x64_py27_1.1.0.tar.bz2", + "17eecb095bd3b0774a87a38624a77ce35e497cd2"), + ('v140', '3.5'):("https://github.com/willyd/caffe-builder/releases/download/v1.1.0/libraries_v140_x64_py35_1.1.0.tar.bz2", + "f060403fd1a7448d866d27c0e5b7dced39c0a607"), } # function for checking SHA1. From a48e81ff722263e0ff3a0792e69c1ccbbc1ec77a Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Sat, 8 Apr 2017 14:12:45 -0400 Subject: [PATCH 47/51] Fixed issue with missing pydot and graphviz under windows --- README.md | 9 +++++---- python/caffe/draw.py | 30 ++++++++++++++++++++++++++++++ scripts/build_win.cmd | 2 +- 3 files changed, 36 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 37311bdcc65..628ef3100b3 100644 --- a/README.md +++ b/README.md @@ -82,13 +82,14 @@ If CUDA is not installed Caffe will default to a CPU_ONLY build. If you have CUD ### Using the Python interface -The recommended Python distribution is Anaconda or Miniconda. To successfully build the python interface you need to install the following packages: +The recommended Python distribution is Anaconda or Miniconda. To successfully build the python interface you need to add the following conda channels: ``` -conda install --yes numpy scipy matplotlib scikit-image pip six +conda config --add channels conda-forge +conda config --add channels willyd ``` -also you will need a protobuf python package that is compatible with pre-built dependencies. This package can be installed this way: +and install the following packages: ``` -conda install --yes --channel willyd protobuf==3.1.0 +conda install --yes cmake ninja numpy scipy protobuf==3.1.0 six scikit-image pyyaml pydotplus graphviz ``` If Python is installed the default is to build the python interface and python layers. If you wish to disable the python layers or the python build use the CMake options `-DBUILD_python_layer=0` and `-DBUILD_python=0` respectively. In order to use the python interface you need to either add the `C:\Projects\caffe\python` folder to your python path of copy the `C:\Projects\caffe\python\caffe` folder to your `site_packages` folder. diff --git a/python/caffe/draw.py b/python/caffe/draw.py index 8411a41d1d4..1e1aec0b925 100644 --- a/python/caffe/draw.py +++ b/python/caffe/draw.py @@ -9,6 +9,7 @@ Caffe. """ +import os from caffe.proto import caffe_pb2 """ @@ -21,6 +22,35 @@ except ImportError: import pydot + +if os.name == 'nt': + # Workaround to find graphviz executables + # with graphviz conda package under windows + + # Monkeypatch the pydot package + pydot_find_graphviz = pydot.graphviz.find_graphviz + + def resolve_graphviz_executables(): + """ + Resolve the graphviz executables by adding a `graphviz` suffix + to folders located on path + """ + # first check if we can find the executables the normal way + progs = pydot_find_graphviz() + if not progs: + directories = os.environ['PATH'].split(';') + suffix = 'graphviz' + progs = {} + for directory in directories: + for exe in ['dot', 'twopi', 'neato', 'circo', 'fdp']: + full_path = os.path.join(directory, suffix, + '{}.exe'.format(exe)) + if os.path.exists(full_path): + progs[exe] = full_path + return progs + + pydot.graphviz.find_graphviz = resolve_graphviz_executables + # Internal layer and blob styles. LAYER_STYLE_DEFAULT = {'shape': 'record', 'fillcolor': '#6495ED', diff --git a/scripts/build_win.cmd b/scripts/build_win.cmd index d7c4750e3fb..80a9dea9239 100644 --- a/scripts/build_win.cmd +++ b/scripts/build_win.cmd @@ -37,7 +37,7 @@ if DEFINED APPVEYOR ( :: Update conda conda update conda -y :: Download other required packages - conda install --yes cmake ninja numpy scipy protobuf==3.1.0 six scikit-image pyyaml + conda install --yes cmake ninja numpy scipy protobuf==3.1.0 six scikit-image pyyaml pydotplus graphviz if ERRORLEVEL 1 ( echo ERROR: Conda update or install failed From 6280643f2e7e5d892a7301d786f140b0df5a71cb Mon Sep 17 00:00:00 2001 From: PENGUINLIONG Date: Fri, 21 Apr 2017 12:04:48 +0800 Subject: [PATCH 48/51] Added GPU arch option. Allow users to specify GPU arch in `build_win.cmd` for convenient cross cmopilation. --- scripts/build_win.cmd | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/scripts/build_win.cmd b/scripts/build_win.cmd index 80a9dea9239..06f06fababf 100644 --- a/scripts/build_win.cmd +++ b/scripts/build_win.cmd @@ -7,6 +7,7 @@ if DEFINED APPVEYOR ( if NOT DEFINED MSVC_VERSION set MSVC_VERSION=14 if NOT DEFINED WITH_NINJA set WITH_NINJA=1 if NOT DEFINED CPU_ONLY set CPU_ONLY=1 + if NOT DEFINED CUDA_ARCH_NAME set CUDA_ARCH_NAME=Auto if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release if NOT DEFINED USE_NCCL set USE_NCCL=0 if NOT DEFINED CMAKE_BUILD_SHARED_LIBS set CMAKE_BUILD_SHARED_LIBS=0 @@ -73,6 +74,9 @@ if DEFINED APPVEYOR ( if NOT DEFINED WITH_NINJA set WITH_NINJA=1 :: Change to 1 to build caffe without CUDA support if NOT DEFINED CPU_ONLY set CPU_ONLY=0 + :: Change to generate CUDA code for one of the following GPU architectures + :: [Fermi Kepler Maxwell Pascal All] + if NOT DEFINED CUDA_ARCH_NAME set CUDA_ARCH_NAME=Auto :: Change to Debug to build Debug. This is only relevant for the Ninja generator the Visual Studio generator will generate both Debug and Release configs if NOT DEFINED CMAKE_CONFIG set CMAKE_CONFIG=Release :: Set to 1 to use NCCL @@ -120,6 +124,7 @@ echo INFO: MSVC_VERSION = !MSVC_VERSION! echo INFO: WITH_NINJA = !WITH_NINJA! echo INFO: CMAKE_GENERATOR = "!CMAKE_GENERATOR!" echo INFO: CPU_ONLY = !CPU_ONLY! +echo INFO: CUDA_ARCH_NAME = !CUDA_ARCH_NAME! echo INFO: CMAKE_CONFIG = !CMAKE_CONFIG! echo INFO: USE_NCCL = !USE_NCCL! echo INFO: CMAKE_BUILD_SHARED_LIBS = !CMAKE_BUILD_SHARED_LIBS! @@ -163,6 +168,7 @@ cmake -G"!CMAKE_GENERATOR!" ^ -DCOPY_PREREQUISITES:BOOL=1 ^ -DINSTALL_PREREQUISITES:BOOL=1 ^ -DUSE_NCCL:BOOL=!USE_NCCL! ^ + -DCUDA_ARCH_NAME:STRING=%CUDA_ARCH_NAME% ^ "%~dp0\.." if ERRORLEVEL 1 ( From c46b9103fa7fbec29fbee8851841336cde7f863e Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Wed, 26 Apr 2017 20:25:30 -0400 Subject: [PATCH 49/51] Added missing dependency between libcaffe and nccl --- src/caffe/CMakeLists.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/caffe/CMakeLists.txt b/src/caffe/CMakeLists.txt index 591a93560d7..eca1d4ec974 100644 --- a/src/caffe/CMakeLists.txt +++ b/src/caffe/CMakeLists.txt @@ -90,6 +90,9 @@ if(MSVC) # Disable Boost autolinking for consuming projects target_compile_definitions(caffe PUBLIC -DBOOST_ALL_NO_LIB) endif() +if(MSVC AND USE_NCCL) + add_dependencies(caffe nccl) +endif() configure_file(${caffe_export_hdr_in} ${caffe_export_hdr}) From ea0d92db2bcf4b8280e809d3e1451e8855867805 Mon Sep 17 00:00:00 2001 From: Guillaume Dumont Date: Thu, 4 May 2017 15:03:07 -0400 Subject: [PATCH 50/51] Added support for resolving MATLAB prerequisites. Also fixes a bug with CMake GetPrerequisites module (see https://gitlab.kitware.com/cmake/cmake/merge_requests/804). --- cmake/CaffeGetPrerequisites.cmake | 1036 ++++++++++++++++++++++++++++++++ cmake/TargetResolvePrerequesites.cmake | 3 +- matlab/CMakeLists.txt | 21 +- 3 files changed, 1055 insertions(+), 5 deletions(-) create mode 100644 cmake/CaffeGetPrerequisites.cmake diff --git a/cmake/CaffeGetPrerequisites.cmake b/cmake/CaffeGetPrerequisites.cmake new file mode 100644 index 00000000000..bf5bc271436 --- /dev/null +++ b/cmake/CaffeGetPrerequisites.cmake @@ -0,0 +1,1036 @@ +# Distributed under the OSI-approved BSD 3-Clause License. See accompanying +# file Copyright.txt or https://cmake.org/licensing for details. + +#.rst: +# GetPrerequisites +# ---------------- +# +# Functions to analyze and list executable file prerequisites. +# +# This module provides functions to list the .dll, .dylib or .so files +# that an executable or shared library file depends on. (Its +# prerequisites.) +# +# It uses various tools to obtain the list of required shared library +# files: +# +# :: +# +# dumpbin (Windows) +# objdump (MinGW on Windows) +# ldd (Linux/Unix) +# otool (Mac OSX) +# +# The following functions are provided by this module: +# +# :: +# +# get_prerequisites +# list_prerequisites +# list_prerequisites_by_glob +# gp_append_unique +# is_file_executable +# gp_item_default_embedded_path +# (projects can override with gp_item_default_embedded_path_override) +# gp_resolve_item +# (projects can override with gp_resolve_item_override) +# gp_resolved_file_type +# (projects can override with gp_resolved_file_type_override) +# gp_file_type +# +# Requires CMake 2.6 or greater because it uses function, break, return +# and PARENT_SCOPE. +# +# :: +# +# GET_PREREQUISITES( +# []) +# +# Get the list of shared library files required by . The list +# in the variable named should be empty on first +# entry to this function. On exit, will contain the +# list of required shared library files. +# +# is the full path to an executable file. +# is the name of a CMake variable to contain the results. +# must be 0 or 1 indicating whether to include or +# exclude "system" prerequisites. If is set to 1 all +# prerequisites will be found recursively, if set to 0 only direct +# prerequisites are listed. is the path to the top level +# executable used for @executable_path replacment on the Mac. is +# a list of paths where libraries might be found: these paths are +# searched first when a target without any path info is given. Then +# standard system locations are also searched: PATH, Framework +# locations, /usr/lib... +# +# :: +# +# LIST_PREREQUISITES( [ [ []]]) +# +# Print a message listing the prerequisites of . +# +# is the name of a shared library or executable target or the +# full path to a shared library or executable file. If is set +# to 1 all prerequisites will be found recursively, if set to 0 only +# direct prerequisites are listed. must be 0 or 1 +# indicating whether to include or exclude "system" prerequisites. With +# set to 0 only the full path names of the prerequisites are +# printed, set to 1 extra informatin will be displayed. +# +# :: +# +# LIST_PREREQUISITES_BY_GLOB( ) +# +# Print the prerequisites of shared library and executable files +# matching a globbing pattern. is GLOB or GLOB_RECURSE and +# is a globbing expression used with "file(GLOB" or +# "file(GLOB_RECURSE" to retrieve a list of matching files. If a +# matching file is executable, its prerequisites are listed. +# +# Any additional (optional) arguments provided are passed along as the +# optional arguments to the list_prerequisites calls. +# +# :: +# +# GP_APPEND_UNIQUE( ) +# +# Append to the list variable only if the value is +# not already in the list. +# +# :: +# +# IS_FILE_EXECUTABLE( ) +# +# Return 1 in if is a binary executable, 0 +# otherwise. +# +# :: +# +# GP_ITEM_DEFAULT_EMBEDDED_PATH( ) +# +# Return the path that others should refer to the item by when the item +# is embedded inside a bundle. +# +# Override on a per-project basis by providing a project-specific +# gp_item_default_embedded_path_override function. +# +# :: +# +# GP_RESOLVE_ITEM( +# []) +# +# Resolve an item into an existing full path file. +# +# Override on a per-project basis by providing a project-specific +# gp_resolve_item_override function. +# +# :: +# +# GP_RESOLVED_FILE_TYPE( +# []) +# +# Return the type of with respect to . String +# describing type of prerequisite is returned in variable named +# . +# +# Use and if necessary to resolve non-absolute +# values -- but only for non-embedded items. +# +# Possible types are: +# +# :: +# +# system +# local +# embedded +# other +# +# Override on a per-project basis by providing a project-specific +# gp_resolved_file_type_override function. +# +# :: +# +# GP_FILE_TYPE( ) +# +# Return the type of with respect to . String +# describing type of prerequisite is returned in variable named +# . +# +# Possible types are: +# +# :: +# +# system +# local +# embedded +# other + +function(gp_append_unique list_var value) + set(contains 0) + + foreach(item ${${list_var}}) + if(item STREQUAL "${value}") + set(contains 1) + break() + endif() + endforeach() + + if(NOT contains) + set(${list_var} ${${list_var}} "${value}" PARENT_SCOPE) + endif() +endfunction() + + +function(is_file_executable file result_var) + # + # A file is not executable until proven otherwise: + # + set(${result_var} 0 PARENT_SCOPE) + + get_filename_component(file_full "${file}" ABSOLUTE) + string(TOLOWER "${file_full}" file_full_lower) + + # If file name ends in .exe on Windows, *assume* executable: + # + if(WIN32 AND NOT UNIX) + if("${file_full_lower}" MATCHES "\\.exe$") + set(${result_var} 1 PARENT_SCOPE) + return() + endif() + + # A clause could be added here that uses output or return value of dumpbin + # to determine ${result_var}. In 99%+? practical cases, the exe name + # match will be sufficient... + # + endif() + + # Use the information returned from the Unix shell command "file" to + # determine if ${file_full} should be considered an executable file... + # + # If the file command's output contains "executable" and does *not* contain + # "text" then it is likely an executable suitable for prerequisite analysis + # via the get_prerequisites macro. + # + if(UNIX) + if(NOT file_cmd) + find_program(file_cmd "file") + mark_as_advanced(file_cmd) + endif() + + if(file_cmd) + execute_process(COMMAND "${file_cmd}" "${file_full}" + RESULT_VARIABLE file_rv + OUTPUT_VARIABLE file_ov + ERROR_VARIABLE file_ev + OUTPUT_STRIP_TRAILING_WHITESPACE + ) + if(NOT file_rv STREQUAL "0") + message(FATAL_ERROR "${file_cmd} failed: ${file_rv}\n${file_ev}") + endif() + + # Replace the name of the file in the output with a placeholder token + # (the string " _file_full_ ") so that just in case the path name of + # the file contains the word "text" or "executable" we are not fooled + # into thinking "the wrong thing" because the file name matches the + # other 'file' command output we are looking for... + # + string(REPLACE "${file_full}" " _file_full_ " file_ov "${file_ov}") + string(TOLOWER "${file_ov}" file_ov) + + #message(STATUS "file_ov='${file_ov}'") + if("${file_ov}" MATCHES "executable") + #message(STATUS "executable!") + if("${file_ov}" MATCHES "text") + #message(STATUS "but text, so *not* a binary executable!") + else() + set(${result_var} 1 PARENT_SCOPE) + return() + endif() + endif() + + # Also detect position independent executables on Linux, + # where "file" gives "shared object ... (uses shared libraries)" + if("${file_ov}" MATCHES "shared object.*\(uses shared libs\)") + set(${result_var} 1 PARENT_SCOPE) + return() + endif() + + # "file" version 5.22 does not print "(used shared libraries)" + # but uses "interpreter" + if("${file_ov}" MATCHES "shared object.*interpreter") + set(${result_var} 1 PARENT_SCOPE) + return() + endif() + + else() + message(STATUS "warning: No 'file' command, skipping execute_process...") + endif() + endif() +endfunction() + + +function(gp_item_default_embedded_path item default_embedded_path_var) + + # On Windows and Linux, "embed" prerequisites in the same directory + # as the executable by default: + # + set(path "@executable_path") + set(overridden 0) + + # On the Mac, relative to the executable depending on the type + # of the thing we are embedding: + # + if(APPLE) + # + # The assumption here is that all executables in the bundle will be + # in same-level-directories inside the bundle. The parent directory + # of an executable inside the bundle should be MacOS or a sibling of + # MacOS and all embedded paths returned from here will begin with + # "@executable_path/../" and will work from all executables in all + # such same-level-directories inside the bundle. + # + + # By default, embed things right next to the main bundle executable: + # + set(path "@executable_path/../../Contents/MacOS") + + # Embed .dylibs right next to the main bundle executable: + # + if(item MATCHES "\\.dylib$") + set(path "@executable_path/../MacOS") + set(overridden 1) + endif() + + # Embed frameworks in the embedded "Frameworks" directory (sibling of MacOS): + # + if(NOT overridden) + if(item MATCHES "[^/]+\\.framework/") + set(path "@executable_path/../Frameworks") + set(overridden 1) + endif() + endif() + endif() + + # Provide a hook so that projects can override the default embedded location + # of any given library by whatever logic they choose: + # + if(COMMAND gp_item_default_embedded_path_override) + gp_item_default_embedded_path_override("${item}" path) + endif() + + set(${default_embedded_path_var} "${path}" PARENT_SCOPE) +endfunction() + + +function(gp_resolve_item context item exepath dirs resolved_item_var) + set(resolved 0) + set(resolved_item "${item}") + if(ARGC GREATER 5) + set(rpaths "${ARGV5}") + else() + set(rpaths "") + endif() + + # Is it already resolved? + # + if(IS_ABSOLUTE "${resolved_item}" AND EXISTS "${resolved_item}") + set(resolved 1) + endif() + + if(NOT resolved) + if(item MATCHES "^@executable_path") + # + # @executable_path references are assumed relative to exepath + # + string(REPLACE "@executable_path" "${exepath}" ri "${item}") + get_filename_component(ri "${ri}" ABSOLUTE) + + if(EXISTS "${ri}") + #message(STATUS "info: embedded item exists (${ri})") + set(resolved 1) + set(resolved_item "${ri}") + else() + message(STATUS "warning: embedded item does not exist '${ri}'") + endif() + endif() + endif() + + if(NOT resolved) + if(item MATCHES "^@loader_path") + # + # @loader_path references are assumed relative to the + # PATH of the given "context" (presumably another library) + # + get_filename_component(contextpath "${context}" PATH) + string(REPLACE "@loader_path" "${contextpath}" ri "${item}") + get_filename_component(ri "${ri}" ABSOLUTE) + + if(EXISTS "${ri}") + #message(STATUS "info: embedded item exists (${ri})") + set(resolved 1) + set(resolved_item "${ri}") + else() + message(STATUS "warning: embedded item does not exist '${ri}'") + endif() + endif() + endif() + + if(NOT resolved) + if(item MATCHES "^@rpath") + # + # @rpath references are relative to the paths built into the binaries with -rpath + # We handle this case like we do for other Unixes + # + string(REPLACE "@rpath/" "" norpath_item "${item}") + + set(ri "ri-NOTFOUND") + find_file(ri "${norpath_item}" ${exepath} ${dirs} ${rpaths} NO_DEFAULT_PATH) + if(ri) + #message(STATUS "info: 'find_file' in exepath/dirs/rpaths (${ri})") + set(resolved 1) + set(resolved_item "${ri}") + set(ri "ri-NOTFOUND") + endif() + + endif() + endif() + + if(NOT resolved) + set(ri "ri-NOTFOUND") + find_file(ri "${item}" ${exepath} ${dirs} NO_DEFAULT_PATH) + find_file(ri "${item}" ${exepath} ${dirs} /usr/lib) + if(ri) + #message(STATUS "info: 'find_file' in exepath/dirs (${ri})") + set(resolved 1) + set(resolved_item "${ri}") + set(ri "ri-NOTFOUND") + endif() + endif() + + if(NOT resolved) + if(item MATCHES "[^/]+\\.framework/") + set(fw "fw-NOTFOUND") + find_file(fw "${item}" + "~/Library/Frameworks" + "/Library/Frameworks" + "/System/Library/Frameworks" + ) + if(fw) + #message(STATUS "info: 'find_file' found framework (${fw})") + set(resolved 1) + set(resolved_item "${fw}") + set(fw "fw-NOTFOUND") + endif() + endif() + endif() + + # Using find_program on Windows will find dll files that are in the PATH. + # (Converting simple file names into full path names if found.) + # + if(WIN32 AND NOT UNIX) + if(NOT resolved) + set(ri "ri-NOTFOUND") + find_program(ri "${item}" PATHS ${exepath} ${dirs} NO_DEFAULT_PATH) + find_program(ri "${item}" PATHS ${exepath} ${dirs}) + if(ri) + #message(STATUS "info: 'find_program' in exepath/dirs (${ri})") + set(resolved 1) + set(resolved_item "${ri}") + set(ri "ri-NOTFOUND") + endif() + endif() + endif() + + # Provide a hook so that projects can override item resolution + # by whatever logic they choose: + # + if(COMMAND gp_resolve_item_override) + gp_resolve_item_override("${context}" "${item}" "${exepath}" "${dirs}" resolved_item resolved) + endif() + + if(NOT resolved) + message(STATUS " +warning: cannot resolve item '${item}' + + possible problems: + need more directories? + need to use InstallRequiredSystemLibraries? + run in install tree instead of build tree? +") +# message(STATUS " +#****************************************************************************** +#warning: cannot resolve item '${item}' +# +# possible problems: +# need more directories? +# need to use InstallRequiredSystemLibraries? +# run in install tree instead of build tree? +# +# context='${context}' +# item='${item}' +# exepath='${exepath}' +# dirs='${dirs}' +# resolved_item_var='${resolved_item_var}' +#****************************************************************************** +#") + endif() + + set(${resolved_item_var} "${resolved_item}" PARENT_SCOPE) +endfunction() + + +function(gp_resolved_file_type original_file file exepath dirs type_var) + if(ARGC GREATER 5) + set(rpaths "${ARGV5}") + else() + set(rpaths "") + endif() + #message(STATUS "**") + + if(NOT IS_ABSOLUTE "${original_file}") + message(STATUS "warning: gp_resolved_file_type expects absolute full path for first arg original_file") + endif() + if(IS_ABSOLUTE "${original_file}") + get_filename_component(original_file "${original_file}" ABSOLUTE) # canonicalize path + endif() + + set(is_embedded 0) + set(is_local 0) + set(is_system 0) + + set(resolved_file "${file}") + + if("${file}" MATCHES "^@(executable|loader)_path") + set(is_embedded 1) + endif() + + if(NOT is_embedded) + if(NOT IS_ABSOLUTE "${file}") + gp_resolve_item("${original_file}" "${file}" "${exepath}" "${dirs}" resolved_file "${rpaths}") + endif() + if(IS_ABSOLUTE "${resolved_file}") + get_filename_component(resolved_file "${resolved_file}" ABSOLUTE) # canonicalize path + endif() + + string(TOLOWER "${original_file}" original_lower) + string(TOLOWER "${resolved_file}" lower) + + if(UNIX) + if(resolved_file MATCHES "^(/lib/|/lib32/|/lib64/|/usr/lib/|/usr/lib32/|/usr/lib64/|/usr/X11R6/|/usr/bin/)") + set(is_system 1) + endif() + endif() + + if(APPLE) + if(resolved_file MATCHES "^(/System/Library/|/usr/lib/)") + set(is_system 1) + endif() + endif() + + if(WIN32) + string(TOLOWER "$ENV{SystemRoot}" sysroot) + file(TO_CMAKE_PATH "${sysroot}" sysroot) + + string(TOLOWER "$ENV{windir}" windir) + file(TO_CMAKE_PATH "${windir}" windir) + + if(lower MATCHES "^(${sysroot}/sys(tem|wow)|${windir}/sys(tem|wow)|(.*/)*(msvc|api-ms-win-)[^/]+dll)") + set(is_system 1) + endif() + + if(UNIX) + # if cygwin, we can get the properly formed windows paths from cygpath + find_program(CYGPATH_EXECUTABLE cygpath) + + if(CYGPATH_EXECUTABLE) + execute_process(COMMAND ${CYGPATH_EXECUTABLE} -W + RESULT_VARIABLE env_rv + OUTPUT_VARIABLE env_windir + ERROR_VARIABLE env_ev + OUTPUT_STRIP_TRAILING_WHITESPACE) + if(NOT env_rv STREQUAL "0") + message(FATAL_ERROR "${CYGPATH_EXECUTABLE} -W failed: ${env_rv}\n${env_ev}") + endif() + execute_process(COMMAND ${CYGPATH_EXECUTABLE} -S + RESULT_VARIABLE env_rv + OUTPUT_VARIABLE env_sysdir + ERROR_VARIABLE env_ev + OUTPUT_STRIP_TRAILING_WHITESPACE) + if(NOT env_rv STREQUAL "0") + message(FATAL_ERROR "${CYGPATH_EXECUTABLE} -S failed: ${env_rv}\n${env_ev}") + endif() + string(TOLOWER "${env_windir}" windir) + string(TOLOWER "${env_sysdir}" sysroot) + + if(lower MATCHES "^(${sysroot}/sys(tem|wow)|${windir}/sys(tem|wow)|(.*/)*(msvc|api-ms-win-)[^/]+dll)") + set(is_system 1) + endif() + endif() + endif() + endif() + + if(NOT is_system) + get_filename_component(original_path "${original_lower}" PATH) + get_filename_component(path "${lower}" PATH) + if(original_path STREQUAL path) + set(is_local 1) + else() + string(LENGTH "${original_path}/" original_length) + string(LENGTH "${lower}" path_length) + if(${path_length} GREATER ${original_length}) + string(SUBSTRING "${lower}" 0 ${original_length} path) + if("${original_path}/" STREQUAL path) + set(is_embedded 1) + endif() + endif() + endif() + endif() + endif() + + # Return type string based on computed booleans: + # + set(type "other") + + if(is_system) + set(type "system") + elseif(is_embedded) + set(type "embedded") + elseif(is_local) + set(type "local") + endif() + + #message(STATUS "gp_resolved_file_type: '${file}' '${resolved_file}'") + #message(STATUS " type: '${type}'") + + if(NOT is_embedded) + if(NOT IS_ABSOLUTE "${resolved_file}") + if(lower MATCHES "^msvc[^/]+dll" AND is_system) + message(STATUS "info: non-absolute msvc file '${file}' returning type '${type}'") + else() + message(STATUS "warning: gp_resolved_file_type non-absolute file '${file}' returning type '${type}' -- possibly incorrect") + endif() + endif() + endif() + + # Provide a hook so that projects can override the decision on whether a + # library belongs to the system or not by whatever logic they choose: + # + if(COMMAND gp_resolved_file_type_override) + gp_resolved_file_type_override("${resolved_file}" type) + endif() + + set(${type_var} "${type}" PARENT_SCOPE) + + #message(STATUS "**") +endfunction() + + +function(gp_file_type original_file file type_var) + if(NOT IS_ABSOLUTE "${original_file}") + message(STATUS "warning: gp_file_type expects absolute full path for first arg original_file") + endif() + + get_filename_component(exepath "${original_file}" PATH) + + set(type "") + gp_resolved_file_type("${original_file}" "${file}" "${exepath}" "" type) + + set(${type_var} "${type}" PARENT_SCOPE) +endfunction() + + +function(get_prerequisites target prerequisites_var exclude_system recurse exepath dirs) + set(verbose 0) + set(eol_char "E") + if(ARGC GREATER 6) + set(rpaths "${ARGV6}") + else() + set(rpaths "") + endif() + + if(NOT IS_ABSOLUTE "${target}") + message("warning: target '${target}' is not absolute...") + endif() + + if(NOT EXISTS "${target}") + message("warning: target '${target}' does not exist...") + set(${prerequisites_var} "" PARENT_SCOPE) + return() + endif() + + set(gp_cmd_paths ${gp_cmd_paths} + "[HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\14.0;InstallDir]/../../VC/bin" + "$ENV{VS140COMNTOOLS}/../../VC/bin" + "C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin" + "[HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\12.0;InstallDir]/../../VC/bin" + "$ENV{VS120COMNTOOLS}/../../VC/bin" + "C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/bin" + "[HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\11.0;InstallDir]/../../VC/bin" + "$ENV{VS110COMNTOOLS}/../../VC/bin" + "C:/Program Files (x86)/Microsoft Visual Studio 11.0/VC/bin" + "[HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\10.0;InstallDir]/../../VC/bin" + "$ENV{VS100COMNTOOLS}/../../VC/bin" + "C:/Program Files (x86)/Microsoft Visual Studio 10.0/VC/bin" + "[HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\9.0;InstallDir]/../../VC/bin" + "$ENV{VS90COMNTOOLS}/../../VC/bin" + "C:/Program Files/Microsoft Visual Studio 9.0/VC/bin" + "C:/Program Files (x86)/Microsoft Visual Studio 9.0/VC/bin" + "[HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\8.0;InstallDir]/../../VC/bin" + "$ENV{VS80COMNTOOLS}/../../VC/bin" + "C:/Program Files/Microsoft Visual Studio 8/VC/BIN" + "C:/Program Files (x86)/Microsoft Visual Studio 8/VC/BIN" + "[HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\VisualStudio\\7.1;InstallDir]/../../VC7/bin" + "$ENV{VS71COMNTOOLS}/../../VC7/bin" + "C:/Program Files/Microsoft Visual Studio .NET 2003/VC7/BIN" + "C:/Program Files (x86)/Microsoft Visual Studio .NET 2003/VC7/BIN" + "/usr/local/bin" + "/usr/bin" + ) + + # + # + # Try to choose the right tool by default. Caller can set gp_tool prior to + # calling this function to force using a different tool. + # + if(NOT gp_tool) + set(gp_tool "ldd") + + if(APPLE) + set(gp_tool "otool") + endif() + + if(WIN32 AND NOT UNIX) # This is how to check for cygwin, har! + find_program(gp_dumpbin "dumpbin" PATHS ${gp_cmd_paths}) + if(gp_dumpbin) + set(gp_tool "dumpbin") + else() # Try harder. Maybe we're on MinGW + set(gp_tool "objdump") + endif() + endif() + endif() + + find_program(gp_cmd ${gp_tool} PATHS ${gp_cmd_paths}) + + if(NOT gp_cmd) + message(STATUS "warning: could not find '${gp_tool}' - cannot analyze prerequisites...") + return() + endif() + + set(gp_cmd_maybe_filter) # optional command to pre-filter gp_tool results + + if(gp_tool STREQUAL "ldd") + set(gp_cmd_args "") + set(gp_regex "^[\t ]*[^\t ]+ => ([^\t\(]+) .*${eol_char}$") + set(gp_regex_error "not found${eol_char}$") + set(gp_regex_fallback "^[\t ]*([^\t ]+) => ([^\t ]+).*${eol_char}$") + set(gp_regex_cmp_count 1) + elseif(gp_tool STREQUAL "otool") + set(gp_cmd_args "-L") + set(gp_regex "^\t([^\t]+) \\(compatibility version ([0-9]+.[0-9]+.[0-9]+), current version ([0-9]+.[0-9]+.[0-9]+)\\)${eol_char}$") + set(gp_regex_error "") + set(gp_regex_fallback "") + set(gp_regex_cmp_count 3) + elseif(gp_tool STREQUAL "dumpbin") + set(gp_cmd_args "/dependents") + set(gp_regex "^ ([^ ].*[Dd][Ll][Ll])${eol_char}$") + set(gp_regex_error "") + set(gp_regex_fallback "") + set(gp_regex_cmp_count 1) + elseif(gp_tool STREQUAL "objdump") + set(gp_cmd_args "-p") + set(gp_regex "^\t*DLL Name: (.*\\.[Dd][Ll][Ll])${eol_char}$") + set(gp_regex_error "") + set(gp_regex_fallback "") + set(gp_regex_cmp_count 1) + # objdump generates copious output so we create a grep filter to pre-filter results + if(WIN32) + find_program(gp_grep_cmd findstr) + else() + find_program(gp_grep_cmd grep) + endif() + if(gp_grep_cmd) + set(gp_cmd_maybe_filter COMMAND ${gp_grep_cmd} "-a" "^[[:blank:]]*DLL Name: ") + endif() + else() + message(STATUS "warning: gp_tool='${gp_tool}' is an unknown tool...") + message(STATUS "CMake function get_prerequisites needs more code to handle '${gp_tool}'") + message(STATUS "Valid gp_tool values are dumpbin, ldd, objdump and otool.") + return() + endif() + + + if(gp_tool STREQUAL "dumpbin") + # When running dumpbin, it also needs the "Common7/IDE" directory in the + # PATH. It will already be in the PATH if being run from a Visual Studio + # command prompt. Add it to the PATH here in case we are running from a + # different command prompt. + # + get_filename_component(gp_cmd_dir "${gp_cmd}" PATH) + get_filename_component(gp_cmd_dlls_dir "${gp_cmd_dir}/../../Common7/IDE" ABSOLUTE) + # Use cmake paths as a user may have a PATH element ending with a backslash. + # This will escape the list delimiter and create havoc! + if(EXISTS "${gp_cmd_dlls_dir}") + # only add to the path if it is not already in the path + set(gp_found_cmd_dlls_dir 0) + file(TO_CMAKE_PATH "$ENV{PATH}" env_path) + foreach(gp_env_path_element ${env_path}) + if(gp_env_path_element STREQUAL gp_cmd_dlls_dir) + set(gp_found_cmd_dlls_dir 1) + endif() + endforeach() + + if(NOT gp_found_cmd_dlls_dir) + file(TO_NATIVE_PATH "${gp_cmd_dlls_dir}" gp_cmd_dlls_dir) + set(ENV{PATH} "$ENV{PATH};${gp_cmd_dlls_dir}") + endif() + endif() + endif() + # + # + + if(gp_tool STREQUAL "ldd") + set(old_ld_env "$ENV{LD_LIBRARY_PATH}") + set(new_ld_env "${exepath}") + foreach(dir ${dirs}) + string(APPEND new_ld_env ":${dir}") + endforeach() + set(ENV{LD_LIBRARY_PATH} "${new_ld_env}:$ENV{LD_LIBRARY_PATH}") + endif() + + + # Track new prerequisites at each new level of recursion. Start with an + # empty list at each level: + # + set(unseen_prereqs) + + # Run gp_cmd on the target: + # + execute_process( + COMMAND ${gp_cmd} ${gp_cmd_args} ${target} + ${gp_cmd_maybe_filter} + RESULT_VARIABLE gp_rv + OUTPUT_VARIABLE gp_cmd_ov + ERROR_VARIABLE gp_ev + ) + + if(gp_tool STREQUAL "dumpbin") + # Exclude delay load dependencies under windows (they are listed in dumpbin output after the message below) + string(FIND "${gp_cmd_ov}" "Image has the following delay load dependencies" gp_delayload_pos) + if (${gp_delayload_pos} GREATER -1) + string(SUBSTRING "${gp_cmd_ov}" 0 ${gp_delayload_pos} gp_cmd_ov_no_delayload_deps) + string(SUBSTRING "${gp_cmd_ov}" ${gp_delayload_pos} -1 gp_cmd_ov_delayload_deps) + if (verbose) + message(STATUS "GetPrequisites(${target}) : ignoring the following delay load dependencies :\n ${gp_cmd_ov_delayload_deps}") + endif() + set(gp_cmd_ov ${gp_cmd_ov_no_delayload_deps}) + endif() + endif() + + if(NOT gp_rv STREQUAL "0") + if(gp_tool STREQUAL "dumpbin") + # dumpbin error messages seem to go to stdout + message(FATAL_ERROR "${gp_cmd} failed: ${gp_rv}\n${gp_ev}\n${gp_cmd_ov}") + else() + message(FATAL_ERROR "${gp_cmd} failed: ${gp_rv}\n${gp_ev}") + endif() + endif() + + if(gp_tool STREQUAL "ldd") + set(ENV{LD_LIBRARY_PATH} "${old_ld_env}") + endif() + + if(verbose) + message(STATUS "") + message(STATUS "gp_cmd_ov='${gp_cmd_ov}'") + message(STATUS "") + endif() + + get_filename_component(target_dir "${target}" PATH) + + # Convert to a list of lines: + # + string(REPLACE ";" "\\;" candidates "${gp_cmd_ov}") + string(REPLACE "\n" "${eol_char};" candidates "${candidates}") + + # check for install id and remove it from list, since otool -L can include a + # reference to itself + set(gp_install_id) + if(gp_tool STREQUAL "otool") + execute_process( + COMMAND otool -D ${target} + RESULT_VARIABLE otool_rv + OUTPUT_VARIABLE gp_install_id_ov + ERROR_VARIABLE otool_ev + ) + if(NOT otool_rv STREQUAL "0") + message(FATAL_ERROR "otool -D failed: ${otool_rv}\n${otool_ev}") + endif() + # second line is install name + string(REGEX REPLACE ".*:\n" "" gp_install_id "${gp_install_id_ov}") + if(gp_install_id) + # trim + string(REGEX MATCH "[^\n ].*[^\n ]" gp_install_id "${gp_install_id}") + #message("INSTALL ID is \"${gp_install_id}\"") + endif() + endif() + + # Analyze each line for file names that match the regular expression: + # + foreach(candidate ${candidates}) + if("${candidate}" MATCHES "${gp_regex}") + + # Extract information from each candidate: + if(gp_regex_error AND "${candidate}" MATCHES "${gp_regex_error}") + string(REGEX REPLACE "${gp_regex_fallback}" "\\1" raw_item "${candidate}") + else() + string(REGEX REPLACE "${gp_regex}" "\\1" raw_item "${candidate}") + endif() + + if(gp_regex_cmp_count GREATER 1) + string(REGEX REPLACE "${gp_regex}" "\\2" raw_compat_version "${candidate}") + string(REGEX REPLACE "^([0-9]+)\\.([0-9]+)\\.([0-9]+)$" "\\1" compat_major_version "${raw_compat_version}") + string(REGEX REPLACE "^([0-9]+)\\.([0-9]+)\\.([0-9]+)$" "\\2" compat_minor_version "${raw_compat_version}") + string(REGEX REPLACE "^([0-9]+)\\.([0-9]+)\\.([0-9]+)$" "\\3" compat_patch_version "${raw_compat_version}") + endif() + + if(gp_regex_cmp_count GREATER 2) + string(REGEX REPLACE "${gp_regex}" "\\3" raw_current_version "${candidate}") + string(REGEX REPLACE "^([0-9]+)\\.([0-9]+)\\.([0-9]+)$" "\\1" current_major_version "${raw_current_version}") + string(REGEX REPLACE "^([0-9]+)\\.([0-9]+)\\.([0-9]+)$" "\\2" current_minor_version "${raw_current_version}") + string(REGEX REPLACE "^([0-9]+)\\.([0-9]+)\\.([0-9]+)$" "\\3" current_patch_version "${raw_current_version}") + endif() + + # Use the raw_item as the list entries returned by this function. Use the + # gp_resolve_item function to resolve it to an actual full path file if + # necessary. + # + set(item "${raw_item}") + + # Add each item unless it is excluded: + # + set(add_item 1) + + if(item STREQUAL gp_install_id) + set(add_item 0) + endif() + + if(add_item AND ${exclude_system}) + set(type "") + gp_resolved_file_type("${target}" "${item}" "${exepath}" "${dirs}" type "${rpaths}") + + if(type STREQUAL "system") + set(add_item 0) + endif() + endif() + + if(add_item) + list(LENGTH ${prerequisites_var} list_length_before_append) + gp_append_unique(${prerequisites_var} "${item}") + list(LENGTH ${prerequisites_var} list_length_after_append) + + if(${recurse}) + # If item was really added, this is the first time we have seen it. + # Add it to unseen_prereqs so that we can recursively add *its* + # prerequisites... + # + # But first: resolve its name to an absolute full path name such + # that the analysis tools can simply accept it as input. + # + if(NOT list_length_before_append EQUAL list_length_after_append) + gp_resolve_item("${target}" "${item}" "${exepath}" "${dirs}" resolved_item "${rpaths}") + if(EXISTS ${resolved_item}) + # Recurse only if we could resolve the item. + # Otherwise the prerequisites_var list will be cleared + set(unseen_prereqs ${unseen_prereqs} "${resolved_item}") + endif() + endif() + endif() + endif() + else() + if(verbose) + message(STATUS "ignoring non-matching line: '${candidate}'") + endif() + endif() + endforeach() + + list(LENGTH ${prerequisites_var} prerequisites_var_length) + if(prerequisites_var_length GREATER 0) + list(SORT ${prerequisites_var}) + endif() + if(${recurse}) + set(more_inputs ${unseen_prereqs}) + foreach(input ${more_inputs}) + get_prerequisites("${input}" ${prerequisites_var} ${exclude_system} ${recurse} "${exepath}" "${dirs}" "${rpaths}") + endforeach() + endif() + + set(${prerequisites_var} ${${prerequisites_var}} PARENT_SCOPE) +endfunction() + + +function(list_prerequisites target) + if(ARGC GREATER 1 AND NOT "${ARGV1}" STREQUAL "") + set(all "${ARGV1}") + else() + set(all 1) + endif() + + if(ARGC GREATER 2 AND NOT "${ARGV2}" STREQUAL "") + set(exclude_system "${ARGV2}") + else() + set(exclude_system 0) + endif() + + if(ARGC GREATER 3 AND NOT "${ARGV3}" STREQUAL "") + set(verbose "${ARGV3}") + else() + set(verbose 0) + endif() + + set(count 0) + set(count_str "") + set(print_count "${verbose}") + set(print_prerequisite_type "${verbose}") + set(print_target "${verbose}") + set(type_str "") + + get_filename_component(exepath "${target}" PATH) + + set(prereqs "") + get_prerequisites("${target}" prereqs ${exclude_system} ${all} "${exepath}" "") + + if(print_target) + message(STATUS "File '${target}' depends on:") + endif() + + foreach(d ${prereqs}) + math(EXPR count "${count} + 1") + + if(print_count) + set(count_str "${count}. ") + endif() + + if(print_prerequisite_type) + gp_file_type("${target}" "${d}" type) + set(type_str " (${type})") + endif() + + message(STATUS "${count_str}${d}${type_str}") + endforeach() +endfunction() + + +function(list_prerequisites_by_glob glob_arg glob_exp) + message(STATUS "=============================================================================") + message(STATUS "List prerequisites of executables matching ${glob_arg} '${glob_exp}'") + message(STATUS "") + file(${glob_arg} file_list ${glob_exp}) + foreach(f ${file_list}) + is_file_executable("${f}" is_f_executable) + if(is_f_executable) + message(STATUS "=============================================================================") + list_prerequisites("${f}" ${ARGN}) + message(STATUS "") + endif() + endforeach() +endfunction() diff --git a/cmake/TargetResolvePrerequesites.cmake b/cmake/TargetResolvePrerequesites.cmake index f58333f7bea..429c113958a 100644 --- a/cmake/TargetResolvePrerequesites.cmake +++ b/cmake/TargetResolvePrerequesites.cmake @@ -1,4 +1,5 @@ set(THIS_FILE ${CMAKE_CURRENT_LIST_FILE}) +set(THIS_DIR ${CMAKE_CURRENT_LIST_DIR}) include(CMakeParseArguments) @@ -171,7 +172,7 @@ endfunction() if(CMAKE_SCRIPT_MODE_FILE) - include(GetPrerequisites) + include(${THIS_DIR}/CaffeGetPrerequisites.cmake) # Recreate a list by replacing the @@ with ; string(REPLACE "@@" ";" DIRECTORIES "${DIRECTORIES}") string(REPLACE "@@" ";" PLUGINS "${PLUGINS}") diff --git a/matlab/CMakeLists.txt b/matlab/CMakeLists.txt index c81c2af8845..25f6a5cbef5 100644 --- a/matlab/CMakeLists.txt +++ b/matlab/CMakeLists.txt @@ -57,10 +57,13 @@ if(build_using MATCHES "Matlab") OUTPUT_NAME caffe_ # change the output name to _caffe.mexw64 LINK_TO caffe # cmake will take care of forwarding the correct transitive library dependencies to your mex file ) - # output the target in the source tree as in the original version. - set_target_properties(matlab PROPERTIES + # output the target in the source tree as in the original version. + set_target_properties(matlab PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${PROJECT_SOURCE_DIR}/matlab/+caffe/private ) + if(COPY_PREREQUISITES) + caffe_copy_prerequisites(matlab DESTINATION ${PROJECT_SOURCE_DIR}/matlab/+caffe/private USE_HARD_LINKS) + endif() else() set(libflags -lcaffe${Caffe_POSTFIX} ${libflags}) # Matlab R2014a complans for -Wl,--whole-archive @@ -86,6 +89,16 @@ elseif(build_using MATCHES "Octave") endif() # ---[ Install -file(GLOB mfiles caffe/*.m) -install(FILES ${mfiles} ${Matlab_caffe_mex} DESTINATION matlab) +if(MSVC) + install(DIRECTORY ${PROJECT_SOURCE_DIR}/matlab DESTINATION . + PATTERN CMakeLists.txt EXCLUDE + PATTERN .gitignore EXCLUDE) +else() + file(GLOB mfiles caffe/*.m) + install(FILES ${mfiles} ${Matlab_caffe_mex} DESTINATION matlab) +endif() + +if(MSVC AND INSTALL_PREREQUISITES) + caffe_install_prerequisites(matlab DESTINATION matlab/+caffe/private) +endif() From ba89321d8f2c2555b9d5e021dc7b91731383e3d9 Mon Sep 17 00:00:00 2001 From: Skylar Downes Date: Thu, 17 Aug 2017 17:57:07 -0700 Subject: [PATCH 51/51] add instructions regarding compiler choice --- README.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 628ef3100b3..e93d03b2c4b 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Windows Caffe -**This is an experimental, communtity based branch led by Guillaume Dumont (@willyd). It is a work-in-progress.** +**This is an experimental, community based branch led by Guillaume Dumont (@willyd). It is a work-in-progress.** This branch of Caffe ports the framework to Windows. @@ -28,6 +28,7 @@ Prebuilt binaries can be downloaded from the latest CI build on appveyor for the ### Requirements - Visual Studio 2013 or 2015 + - Technically only the VS C/C++ compiler is required (cl.exe) - [CMake](https://cmake.org/) 3.4 or higher (Visual Studio and [Ninja](https://ninja-build.org/) generators are supported) ### Optional Dependencies @@ -49,7 +50,17 @@ C:\Projects\caffe> git checkout windows :: Edit any of the options inside build_win.cmd to suit your needs C:\Projects\caffe> scripts\build_win.cmd ``` -The `build_win.cmd` script will download the dependencies, create the Visual Studio project files (or the ninja build files) and build the Release configuration. By default all the required DLLs will be copied (or hard linked when possible) next to the consuming binaries. If you wish to disable this option, you can by changing the command line option `-DCOPY_PREREQUISITES=0`. The prebuilt libraries also provide a `prependpath.bat` batch script that can temporarily modify your `PATH` envrionment variable to make the required DLLs available. +The `build_win.cmd` script will download the dependencies, create the Visual Studio project files (or the ninja build files) and build the Release configuration. By default all the required DLLs will be copied (or hard linked when possible) next to the consuming binaries. If you wish to disable this option, you can by changing the command line option `-DCOPY_PREREQUISITES=0`. The prebuilt libraries also provide a `prependpath.bat` batch script that can temporarily modify your `PATH` environment variable to make the required DLLs available. + +If you have GCC installed (e.g. through MinGW), then Ninja will detect it before detecting the Visual Studio compiler, causing errors. In this case you have several options: + +- [Pass CMake the path](https://cmake.org/Wiki/CMake_FAQ#How_do_I_use_a_different_compiler.3F) (Set `CMAKE_C_COMPILER=your/path/to/cl.exe` and `CMAKE_CXX_COMPILER=your/path/to/cl.exe`) +- or Use the Visual Studio Generator by setting `WITH_NINJA` to 0 (This is slower, but may work even if Ninja is failing.) +- or uninstall your copy of GCC + +The path to cl.exe is usually something like +`"C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/your_processor_architecture/cl.exe".` +If you don't want to install Visual Studio, Microsoft's C/C++ compiler [can be obtained here](http://landinghub.visualstudio.com/visual-cpp-build-tools). Below is a more complete description of some of the steps involved in building caffe. @@ -112,7 +123,7 @@ When working with ninja you don't have the Visual Studio solutions as ninja is m ### Building a shared library -CMake can be used to build a shared library instead of the default static library. To do so follow the above procedure and use `-DBUILD_SHARED_LIBS=ON`. Please note however, that some tests (more specifically the solver related tests) will fail since both the test exectuable and caffe library do not share static objects contained in the protobuf library. +CMake can be used to build a shared library instead of the default static library. To do so follow the above procedure and use `-DBUILD_SHARED_LIBS=ON`. Please note however, that some tests (more specifically the solver related tests) will fail since both the test executable and caffe library do not share static objects contained in the protobuf library. ### Troubleshooting