GlusterFS Performance Tuning

前言

GlusterFS 是一套OpenSource的Distributed File System. 簡單來說，他可以把多台多個儲存資源串連起來變成一個或多個Pool。每一個Pool支援Global Namespace/Common Namespace的特性，也就是說，在不同地方掛載同一個Pool看到的會是相同的東西。

有使用過GlusterFS的人都會發現，以Default的Configuration來跑GlusterFS將會得到慘不忍睹的笑能。因此在這邊分享及驗證透過修改GlusterFS不同的參數對於效能的幫助

Configurations

修改的參數我把他分為三類，分別是IO，Cache和Network的參數修改。

I/O 

io-thread-count (16-> 64)

最多同時IO的Thread個數。數字越大就允許越多同時的R/W。特別是當你有RAID的時候可以調高這參數。Default值是16

write-behind-window-size(1MB-> 1GB)

per-file的write behind buffer. Default值是1MB。

Cache 

cache-size (32MB-> 2GB)

read cache的Size。Default值是32MB

cache-max-file-size (-> 16384PB)

設定最多file cached的大小。

cache-min-file-size (-> 0)

cached file最小的大小。

Network

auto tuning(-> disable)

OS會根據網路的throughput來動態調整TCP的window size。但是如果是在穩定的Lan環境這功能是用不到的，因此關閉他來避免奇怪的問題。

Performance Testing

上述列了幾個Tuning的參數，接下來實際針對每個參數的調整來驗證改善的成效。

Experiment Environment and Method

實驗的環境是兩台Server串成一個GlusterFS cluster。 然後在一台Server mount起來，利用iozone來驗證效能。三台的規格都一樣如下圖。

Experiment Configuration

重新歸納一下調整個參數並且給每一個case做編號。

Experiment Result

Write

從寫入的結果來看，在調整了write-behind-buffer後效能有得到非常大的改善。而理所當然的，record size會影響整個throughput。在record=1024KB的時候有較好的結果。

ReWrite

在rewrite的實驗和write有相近的結果。一樣是透過調整write-behind-buffer 來得到效能的改善。

Read

在Read測試當中，可以發現在record size比較小的時候，關閉tso, gro和gso會有較快的效能。不過這個結果應該與網卡及driver有關。

Re-read

Reread與read有相近的結果。在大record size時一樣能得到較好的throughput。

Conclusion

write-behind-buffer很明顯的改善了寫入的速度。我想這就跟在NFS sync/async的差別一樣吧。

在這次的實驗中，沒有RAID 卡也沒有做multi-IO的測試。猜想應該是這個原因才看不出io-thread-count的效果。

Delete Symantec Netbackup Disk Pool

The process is :

1. Expire all backup image
2. Delete all image fragment
3. Delete Disk Pool

So fist, issue bpimmedia to get  backup_id
# bpimmedia -L -client [client name]

and then expire the image
# bpexpdate -backupid [backupid] -d 0

Second, delete all image fragment by issue nbdelete
# nbdelete --allvolumes -force

Finally, delete disk pool from Administration console or Administrator JAVA console

No Dropbox icon in Ubuntu 13.10

1. Install package
# sudo apt-get install libappindicator1
2.  Log out, and then log back in (or issue dropbox stop && dropbox start at the command line)

REF

Linux Kernel Performance Tuning and Hardening

[Security]Deactivates automatic answers to ICMP broadcasts and protects against smurf attacks.

net.ipv4.icmp_echo_ignore_broadcasts=1

[Security]Enable source validation by reversed path, as specified in RFC1812

/*
0 - No source validation.
1 - Strict mode as defined in RFC3704 Strict Reverse Path. 
    Each incoming packet is tested against the FIB and if the interface
    is not the best reverse path the packet check will fail. 
    By default failed packets are discarded.
2 - Loose mode as defined in RFC3704 Loose Reverse Path. 
    Each incoming packet's source address is also tested against the FIB
    and if the source address is not reachable via any interface the packet check will fail.
*/
net.ipv4.conf.all.rp_filter = 1

[Performance]The number of possible inotify(7) watches

fs.inotify.max_user_watches = 65536

[Tuning]Avoid deleting secondary IPs on deleting the primary IP.

/*
If you remove the primary IP address, all secondary addresses are purged by default as well. 
*/
net.ipv4.conf.default.promote_secondaries = 1
net.ipv4.conf.all.promote_secondaries = 1"

[Security]Not allow source route

net.ipv4.conf.default.accept_source_route = 0

[Tuning]Disable listen for router advertisements in order to choose an IPV6 address and router

net.ipv6.conf.all.autoconf = 0
net.ipv6.conf.all.accept_ra = 0

[Security]Do not allow redirect

net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0

[Tuning]Avoid tcp timeout

net.ipv4.tcp_timestamps = 0

[Performance]Maximum amount for the receive socket memory

net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.ipv4.tcp_rmem = 32768 262144 33554432
net.ipv4.tcp_wmem = 32768 262144 33554432

[Performance]Max open file

fs.file-max = 200000

[Performance]The number of IPC message queue resources allowed

kernel.msgmni = 1024

[Performance]How many semaphore arrays, semaphores can be allocate and how many semaphores can be in a single semaphore array

kernel.sem = 400 307200 128 1024

[Performance]Keepalive time

net.ipv4.tcp_keepalive_time = 900
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 10

[Performance] Page cache limit.

/*Some app use large amounts of memory for accelerated access to business data
Parts of this memory are seldom accessed. When a user request then needs to
access paged out memory, the response time is poor. It is even worse, when an
SAP solution running on Java incurs a Java garbage collection. The system
starts heavy page-in (disc I/O) activity and has a poor response time for an
extended period of time.*/
vm.pagecache_limit_mb = 41000

Max number of open file

* hard nofile 32768
* soft nofile 32768

Setup git server

以前在使用git的時候都是同事幫忙架設，這次剛好有機會順便紀錄一下

Git Server

1. 安裝必要套件

#  apt-get install git-core

2. 建立repository

假設所有的project都欲放置/opt/git 並且建立名為project1的git
a. # mkdir /opt/git
b. # mkdir /opt/git/project1
    # cd /opt/git/project1
    # git --bare init

Git Client

在使用者端，假設project的位置是~/projectA

# mkdir ~/projectA

# cd ~/projectA

# git init

# echo "project init">>init

# git add .

# git commit -m "init project"

# git remote add origin user@[IP]:/opt/git/project1

# git push origin master

iSCSI Target and Initiator

iSCSI Target是Server端，提供iSCSI 空間。
iSCSI Initiator是Client端，連接既有的iSCSI空間來使用。

iSCSI Target

在Ubuntu上安裝

1. 選擇要提供空間的裝置

提供服務的空間可以是一個硬碟、一個partition、一個LVM的LV或是使用DD產生一個空間。
使用以下的方法可以在/media下產生一個20G名稱為volume0的空間。

#  dd if=/dev/zero of=/media/volume0 count=0 obs=1 seek=20G

2. 安裝必要套件

使用APT來安裝必要套件

# apt-get install  iscsitarget iscsitarget-dkms

3. 設定iSCSI 

a. 設定開機自動啟動

Edit /etc/default/iscsitarget

change 

ISCSITARGET_ENABLE=false

to 

ISCSITARGET_ENABLE=true

b. 設定iSCSI Target

Edit /etc/iet/ietd.conf

加入

Target iqn.2010-12.nl.ytec.arbiter:arbiter.blabla.lun1

Lun 0 Path=/media/volume0,Type=fileio,ScsiId=lun0,ScsiSN=lun0

其中，需要修改的參數是:

I. Target 後面接的名字，可自訂。

II. Lun一定要從0開始。

III. PATH後面接的參數是空間的位置，如果是剛剛使用dd 產生出來的則為/media/volume0否則更改為實際的空間例如/dev/sdb1 etc.

4. 重新啟動iSCSI Target

# /etc/init.d/iscsitarget restart

iSCSI Initiator

以OpenSuse為例

1. 安裝必要套件

# yast -i open-iscsi

2. 搜尋可用iSCSI Lun

使用iscsiadm來搜尋可用的裝置

# iscsiadm -m discovery --type=st --portal=[iscsi target的IP]

sample output:

192.168.1.101:3260,1 iqn.2010-12.nl.ytec.arbiter:arbiter.blabla.lun1

3. 連接裝置

確定找的到裝置之後就可以連接iSCSI裝置

# iscsiadm  -m  node  -T iqn.2010-12.nl.ytec.arbiter:arbiter.blabla.lun1 --login

使用lsscsi來檢查是否有連接成功

# lsscsi

Sample output:

[0:2:0:0]    disk    Intel    MegaSR           1.0   /dev/sda

[1:2:0:0]    disk    Intel    RMS25CB080       3.22  /dev/sdb

[18:0:0:0]   disk    IET      VIRTUAL-DISK     0     /dev/sdc

4. 掛載裝置

假設我們欲將裝置format成XFS並掛載到/opt/iscsi

# mkfs.xfs /dev/sdc

# mkdir /opt/iscsi

# mount -t xfs /dev/sdc /opt/iscsi

大資料資料庫儲存-NoSQL比較及效能差異(VII)

在大資料資料庫儲存的最終篇將比較各NoSQL的差異和效能的比較。

NoSQL 比較

Experiment Environment

Hardware and Network Topology

Software Version

MySQL VS Big Data

 Insert Data

 Update Data

Look-up Data

Delete Data