GPU

Rest Term::Tech Note

GPU

LinuxでのGPU利用について整理します。

環境

OS: Ubuntu 16.04 LTS / Amazon Linux AMI release 2017.03
GPU: NVIDIA Geforce GTX 760, 970 / Tesla-K80
CUDA Toolkit 7.5 / 8.0

• deviceQuery ## GTX 760 ./deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 1 CUDA Capable device(s) Device 0: "GeForce GTX 760" CUDA Driver Version / Runtime Version 7.5 / 7.5 CUDA Capability Major/Minor version number: 3.0 Total amount of global memory: 2048 MBytes (2147287040 bytes) ( 6) Multiprocessors, (192) CUDA Cores/MP: 1152 CUDA Cores GPU Max Clock rate: 1084 MHz (1.08 GHz) Memory Clock rate: 3004 Mhz Memory Bus Width: 256-bit L2 Cache Size: 524288 bytes Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096) Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 49152 bytes Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 2048 Maximum number of threads per block: 1024 Max dimension size of a thread block (x,y,z): (1024, 1024, 64) Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535) Maximum memory pitch: 2147483647 bytes Texture alignment: 512 bytes Concurrent copy and kernel execution: Yes with 1 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Disabled Device supports Unified Addressing (UVA): Yes Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime Version = 7.5, NumDevs = 1, Device0 = GeForce GTX 760 Result = PASS ## GTX 970 ./deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 1 CUDA Capable device(s) Device 0: "GeForce GTX 970" CUDA Driver Version / Runtime Version 8.0 / 8.0 CUDA Capability Major/Minor version number: 5.2 Total amount of global memory: 4069 MBytes (4266459136 bytes) (13) Multiprocessors, (128) CUDA Cores/MP: 1664 CUDA Cores GPU Max Clock rate: 1253 MHz (1.25 GHz) Memory Clock rate: 3505 Mhz Memory Bus Width: 256-bit L2 Cache Size: 1835008 bytes Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096) Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 49152 bytes Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 2048 Maximum number of threads per block: 1024 Max dimension size of a thread block (x,y,z): (1024, 1024, 64) Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535) Maximum memory pitch: 2147483647 bytes Texture alignment: 512 bytes Concurrent copy and kernel execution: Yes with 2 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Disabled Device supports Unified Addressing (UVA): Yes Device PCI Domain ID / Bus ID / location ID: 0 / 1 / 0 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = GeForce GTX 970 ## Tesla-K80 ./deviceQuery Starting... CUDA Device Query (Runtime API) version (CUDART static linking) Detected 1 CUDA Capable device(s) Device 0: "Tesla K80" CUDA Driver Version / Runtime Version 8.0 / 8.0 CUDA Capability Major/Minor version number: 3.7 Total amount of global memory: 11440 MBytes (11995578368 bytes) (13) Multiprocessors, (192) CUDA Cores/MP: 2496 CUDA Cores GPU Max Clock rate: 824 MHz (0.82 GHz) Memory Clock rate: 2505 Mhz Memory Bus Width: 384-bit L2 Cache Size: 1572864 bytes Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096) Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers Total amount of constant memory: 65536 bytes Total amount of shared memory per block: 49152 bytes Total number of registers available per block: 65536 Warp size: 32 Maximum number of threads per multiprocessor: 2048 Maximum number of threads per block: 1024 Max dimension size of a thread block (x,y,z): (1024, 1024, 64) Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535) Maximum memory pitch: 2147483647 bytes Texture alignment: 512 bytes Concurrent copy and kernel execution: Yes with 2 copy engine(s) Run time limit on kernels: No Integrated GPU sharing Host Memory: No Support host page-locked memory mapping: Yes Alignment requirement for Surfaces: Yes Device has ECC support: Enabled Device supports Unified Addressing (UVA): Yes Device PCI Domain ID / Bus ID / location ID: 0 / 0 / 30 Compute Mode: < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) > deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = Tesla K80

• bandwidthTest ## GTX 760 [CUDA Bandwidth Test] - Starting... Running on... Device 0: GeForce GTX 760 Quick Mode Host to Device Bandwidth, 1 Device(s) PINNED Memory Transfers Transfer Size (Bytes) Bandwidth(MB/s) 33554432 6589.7 Device to Host Bandwidth, 1 Device(s) PINNED Memory Transfers Transfer Size (Bytes) Bandwidth(MB/s) 33554432 6431.5 Device to Device Bandwidth, 1 Device(s) PINNED Memory Transfers Transfer Size (Bytes) Bandwidth(MB/s) 33554432 146365.2 Result = PASS NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled. ## GTX 970 [CUDA Bandwidth Test] - Starting... Running on... Device 0: GeForce GTX 970 Quick Mode Host to Device Bandwidth, 1 Device(s) PINNED Memory Transfers Transfer Size (Bytes) Bandwidth(MB/s) 33554432 6596.5 Device to Host Bandwidth, 1 Device(s) PINNED Memory Transfers Transfer Size (Bytes) Bandwidth(MB/s) 33554432 6572.5 Device to Device Bandwidth, 1 Device(s) PINNED Memory Transfers Transfer Size (Bytes) Bandwidth(MB/s) 33554432 144692.5 Result = PASS NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled. ## Tesla-K80 [CUDA Bandwidth Test] - Starting... Running on... Device 0: Tesla K80 Quick Mode Host to Device Bandwidth, 1 Device(s) PINNED Memory Transfers Transfer Size (Bytes) Bandwidth(MB/s) 33554432 8934.9 Device to Host Bandwidth, 1 Device(s) PINNED Memory Transfers Transfer Size (Bytes) Bandwidth(MB/s) 33554432 10779.4 Device to Device Bandwidth, 1 Device(s) PINNED Memory Transfers Transfer Size (Bytes) Bandwidth(MB/s) 33554432 156315.7 Result = PASS

nvidia-smi

lオプションくらいしか使ったことなかったけど、モニタリング系オプションとかいろいろあるので活用しよう。

• pmon $ nvidia-smi pmon -i 0 # device id を指定 # gpu pid type sm mem enc dec command # Idx # C/G % % % % name 0 - - - - - - - 0 - - - - - - - 0 5299 C 28 12 0 0 gpu_kmean 0 5299 C 93 63 0 0 gpu_kmean 0 5299 C 92 62 0 0 gpu_kmean 0 5299 C 94 61 0 0 gpu_kmean 0 5299 C 94 63 0 0 gpu_kmean 0 5299 C 93 63 0 0 gpu_kmean 0 5299 C 92 59 0 0 gpu_kmean 0 5299 C 93 63 0 0 gpu_kmean 0 5299 C 92 62 0 0 gpu_kmean

• dmon $ nvidia-smi dmon -i 0 -s pucvmet # gpu pwr temp sm mem enc dec mclk pclk pviol tviol fb bar1 sbecc dbecc pci rxpci txpci # Idx W C % % % % MHz MHz % bool MB MB errs errs errs MB/s MB/s 0 16 47 0 0 0 0 324 135 0 0 1 2 - - 0 0 0 0 15 47 0 0 0 0 324 135 0 0 1 2 - - 0 0 0 0 100 50 49 21 0 0 3004 1341 0 0 840 2 - - 0 4835 431 0 188 59 100 62 0 0 3004 1328 0 0 840 2 - - 0 4784 568 0 156 59 100 79 0 0 3004 1341 0 0 840 2 - - 0 4815 442 0 188 61 100 79 0 0 3004 1341 0 0 840 2 - - 0 4825 500 0 144 59 90 65 0 0 3004 1341 0 0 840 2 - - 0 4812 501 0 186 62 87 49 0 0 3004 1341 0 0 840 2 - - 0 5203 494 0 153 59 100 70 0 0 3004 1328 0 0 840 2 - - 0 4827 505 0 180 63 88 51 0 0 3004 1341 0 0 840 2 - - 0 4822 438 0 149 62 100 78 0 0 3004 1341 0 0 840 2 - - 0 5921 446 0 168 63 83 57 0 0 3004 1328 0 0 840 2 - - 0 4832 438 0 156 63 100 78 0 0 3004 1328 0 0 840 2 - - 0 6575 599 0 169 64 89 51 0 0 3004 1328 0 0 840 2 - - 0 4826 440

• query-gpu $ nvidia-smi -l 1 --query-gpu=memory.total,memory.free,memory.used --format=csv memory.total [MiB], memory.free [MiB], memory.used [MiB] 4068 MiB, 4067 MiB, 1 MiB 4068 MiB, 4067 MiB, 1 MiB 4068 MiB, 3228 MiB, 840 MiB 4068 MiB, 3228 MiB, 840 MiB 4068 MiB, 3228 MiB, 840 MiB

ハードウェア確認

• lshw
  lshwコマンドが便利。lspciをgrepするよりスマートかと。

$ sudo lshw -class diplay *-display 詳細: VGA compatible controller 製品: GK104 [GeForce GTX 760] ベンダー: NVIDIA Corporation 物理ID: 0 バス情報: pci@0000:01:00.0 バージョン: a1 幅: 64 bits クロック: 33MHz 性能: pm msi pciexpress vga_controller bus_master cap_list rom 設定: driver=nvidia latency=0 リソース: irq:16 メモリー:f6000000-f6ffffff メモリー:e0000000-e7ffffff メモリー:e8000000-e9ffffff IOポート:e000(サイズ=128) メモリー:f7000000-f707ffff *-display 詳細: VGA compatible controller 製品: 2nd Generation Core Processor Family Integrated Graphics Controller ベンダー: Intel Corporation 物理ID: 2 バス情報: pci@0000:00:02.0 バージョン: 09 幅: 64 bits クロック: 33MHz 性能: msi pm vga_controller bus_master cap_list rom 設定: driver=i915 latency=0 リソース: irq:50 メモリー:f7400000-f77fffff メモリー:d0000000-dfffffff IOポート:f000(サイズ=64) ## ショートバージョン $ sudo lshw -short -class display H/W path デバイス クラス 詳細 =================================================== /0/100/1/0 display GK104 [GeForce GTX 760] /0/100/2 display 2nd Generation Core Processor Family Integrated Graphics Controller

環境構築関連

• NVIDIAドライバのインストール

  # nouveau 無効化
  # /etc/modprobe.d/blacklist-nouveau.conf に以下を記述
  blacklist nouveau
  blacklist lbm-nouveau
  options nouveau modeset=0
  alias nouveau off
  alias lbm-nouveau off
  
  # /etc/modprobe.d/nouveau-kms.conf に以下を記述
  options nouveau modeset=0
  
  $ sudo update-initramfs -u
  $ sudo reboot
  
  # ドライバインストール
  $ sudo apt-get install linux-image-extra-virtual linux-source linux-headers-`uname -r`
  # NVIDA公式サイトからドライバ(runファイル)を取得しておく
  $ sudo sh NVIDIA-Linux-x86_64-xxx.xx.run -a --disable-nouveau
  $ sudo reboot

• 補助電源挿し忘れ
  グラフィックボードに補助電源を挿し忘れていると、nvidia-xxxコマンド実行時に以下のようなメッセージが出る。知らなかった。

  Unable to determine the device handle for GPU 0000:01:00.0: Unable to communicate with GPU because it is  insufficiently powered.
  This may be because not all required external power cables are
  attached, or the attached cables are not seated properly.

• デスクトップマネージャーの停止
  ドライバインストール前に停止しておくこと。

  $ sudo service lightdm stop

• Nvidiaドライバ(PPA)のインストール

  $ sudo add-apt-repository ppa:graphics-drivers/ppa
  $ sudo apt-get update
  # 以下 xxx の箇所はドライバのバージョン番号を指定
  $ sudo apt-get install nvidia-xxx
  # 以下は不要かも
  $ sudo apt-get install mesa-common-dev
  $ sudo apt-get install freeglut3-dev

OpenCV CUDAサポート関連

お手持ちのNvidia製ビデオカードのGPUアーキテクチャは以下のサイトで確認できる。

• CUDA GPUs

OpenCVビルド時のCMakeオプション例を載せる。自分の環境で利用しているGPUのアーキテクチャ番号を CUDA_ARCH_BIN / CUDA_ARCH_PTX オプションに指定しておく。こうするとビルド時間を大幅に短縮できる。また、僕の環境だとcublasを認識させるには WITH_CUBLAS オプションも明示的指定する必要があった。

cmake ... 省略
 -D CUDA_ARCH_BIN=3.0 \
 -D CUDA_ARCH_PTX=3.0 \
 -D WITH_CUBLAS=ON \

cmake実行結果の該当箇所のみ抜粋

--   NVIDIA CUDA
--     Use CUFFT:                   YES
--     Use CUBLAS:                  YES
--     USE NVCUVID:                 NO
--     NVIDIA GPU arch:             30
--     NVIDIA PTX archs:            30
--     Use fast math:               NO

MXNet

• 環境
  CPU: Intel Core-i5 2500K
  RAM: CFD elixir DDR3-1333 8GB
  SSD: Crucial M4 64GB
  GPU: Nvidia Geforce GTX 760
  CUDA: 8.0, cuDNN 5.1

学習速度メモ

CIFAR-10データセットをResNet 110層(バッチサイズは128で固定)で学習
学習完了まで待つのは大変なので秒間辺りの処理枚数を見る

シングルマシン GPU利用 / cuDNN無し

INFO:root:Epoch[37] Validation-accuracy=0.871044
INFO:root:Epoch[38] Batch [20]  Speed: 72.25 samples/sec        Train-accuracy=0.883203
INFO:root:Epoch[38] Batch [40]  Speed: 68.65 samples/sec        Train-accuracy=0.874219
INFO:root:Epoch[38] Batch [60]  Speed: 68.64 samples/sec        Train-accuracy=0.878906
INFO:root:Epoch[38] Batch [80]  Speed: 68.65 samples/sec        Train-accuracy=0.877344
INFO:root:Epoch[38] Batch [100] Speed: 68.64 samples/sec        Train-accuracy=0.877734

シングルマシン GPU利用 / cuDNN有り

INFO:root:Epoch[39] Batch [40]  Speed: 273.99 samples/sec       Train-accuracy=0.875781
INFO:root:Epoch[39] Batch [60]  Speed: 274.03 samples/sec       Train-accuracy=0.875781
INFO:root:Epoch[39] Batch [80]  Speed: 274.08 samples/sec       Train-accuracy=0.863281
INFO:root:Epoch[39] Batch [100] Speed: 273.97 samples/sec       Train-accuracy=0.871094
INFO:root:Epoch[39] Batch [120] Speed: 273.83 samples/sec       Train-accuracy=0.878906

cuDNNの有無でこんなに速度が変わるとは、。

学習結果

INFO:root:Epoch[299] Batch [20] Speed: 288.44 samples/sec       Train-accuracy=0.992188
INFO:root:Epoch[299] Batch [40] Speed: 273.97 samples/sec       Train-accuracy=0.996094
INFO:root:Epoch[299] Batch [60] Speed: 273.90 samples/sec       Train-accuracy=0.995703
INFO:root:Epoch[299] Batch [80] Speed: 274.26 samples/sec       Train-accuracy=0.992969
INFO:root:Epoch[299] Batch [100]        Speed: 274.14 samples/sec       Train-accuracy=0.992969
INFO:root:Epoch[299] Batch [120]        Speed: 274.10 samples/sec       Train-accuracy=0.996875
INFO:root:Epoch[299] Batch [140]        Speed: 274.09 samples/sec       Train-accuracy=0.996875
INFO:root:Epoch[299] Batch [160]        Speed: 273.77 samples/sec       Train-accuracy=0.993359
INFO:root:Epoch[299] Batch [180]        Speed: 273.88 samples/sec       Train-accuracy=0.996484
INFO:root:Epoch[299] Batch [200]        Speed: 273.95 samples/sec       Train-accuracy=0.994922
INFO:root:Epoch[299] Batch [220]        Speed: 273.88 samples/sec       Train-accuracy=0.995703
INFO:root:Epoch[299] Batch [240]        Speed: 273.88 samples/sec       Train-accuracy=0.996484
INFO:root:Epoch[299] Batch [260]        Speed: 273.95 samples/sec       Train-accuracy=0.994141
INFO:root:Epoch[299] Batch [280]        Speed: 273.66 samples/sec       Train-accuracy=0.996484
INFO:root:Epoch[299] Batch [300]        Speed: 273.73 samples/sec       Train-accuracy=0.996094
INFO:root:Epoch[299] Batch [320]        Speed: 273.67 samples/sec       Train-accuracy=0.994531
INFO:root:Epoch[299] Batch [340]        Speed: 273.77 samples/sec       Train-accuracy=0.992578
INFO:root:Epoch[299] Batch [360]        Speed: 273.65 samples/sec       Train-accuracy=0.994531
INFO:root:Epoch[299] Batch [380]        Speed: 273.64 samples/sec       Train-accuracy=0.994141
INFO:root:Epoch[299] Resetting Data Iterator
INFO:root:Epoch[299] Time cost=181.939
INFO:root:Saved checkpoint to "models/cifar10-0300.params"
INFO:root:Epoch[299] Validation-accuracy=0.934039

ResNet 110層でCIFAR-10を学習、分類精度 93.4%

Faiss

コンパイルログ

$ make -j4 g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c hamming.cpp -o hamming.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c utils.cpp -o utils.o -DFINTEGER=int g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c IndexFlat.cpp -o IndexFlat.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c IndexIVF.cpp -o IndexIVF.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c IndexLSH.cpp -o IndexLSH.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c IndexPQ.cpp -o IndexPQ.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c IndexIVFPQ.cpp -o IndexIVFPQ.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c Clustering.cpp -o Clustering.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c Heap.cpp -o Heap.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c VectorTransform.cpp -o VectorTransform.o -DFINTEGER=int g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c index_io.cpp -o index_io.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c PolysemousTraining.cpp -o PolysemousTraining.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c MetaIndexes.cpp -o MetaIndexes.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c Index.cpp -o Index.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c ProductQuantizer.cpp -o ProductQuantizer.o -DFINTEGER=int MetaIndexes.cpp:199:6: warning: ‘void faiss::{anonymous}::translate_labels(long int, faiss::{anonymous}::idx_t*, long int)’ defined but not used [-Wunused-function] void translate_labels (long n, idx_t *labels, long translation) ^ g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c AutoTune.cpp -o AutoTune.o g++ -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -c AuxIndexStructures.cpp -o AuxIndexStructures.o ar r libfaiss.a hamming.o utils.o IndexFlat.o IndexIVF.o IndexLSH.o IndexPQ.o IndexIVFPQ.o Clustering.o Heap.o VectorTransform.o index_io.o PolysemousTraining.o MetaIndexes.o Index.o ProductQuantizer.o AutoTune.o AuxIndexStructures.o ar: libfaiss.a を作成しています g++ -o tests/demo_ivfpq_indexing -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp tests/demo_ivfpq_indexing.cpp libfaiss.a -g -fPIC -fopenmp /usr/lib/libopenblas.so.0 $ make py g++ -I. -fPIC -m64 -Wall -g -O3 -msse4 -mpopcnt -fopenmp -Wno-sign-compare -Dnullptr=NULL -Doverride= -fopenmp -g -fPIC -fopenmp -I/usr/include/python3.5m -I/usr/local/lib/python3.5/dist-packages/numpy/core/include -shared \ -o python/_swigfaiss.so python/swigfaiss_wrap.cxx libfaiss.a /usr/lib/libopenblas.so.0 cp python/_swigfaiss.so python/swigfaiss.py . ## GPU version $ cd gpu $ make -j4