kzalloc()

• static  inline  void  *kzalloc(size_t size, gfp_t flags): allocate memory. The memory is set to zero.
• Flag: 

           size: 欲申請大小
           flags: same as kmalloc, see kmalloc



static inline void *kzalloc(size_t size, gfp_t flags)
{
return kmalloc(size, flags | __GFP_ZERO);
}

Reference:

include/linux/slab.h

kmalloc()

• void * kmalloc (size_t size, int flags) : 在kernel配置記憶體位置
• INCLUDE    :include/linux/slab.h
• FLAG: 

          size:欲分配大小
          flags:

Action Modifiers( 通常使用後面提到的type modifier)    

__GFP_WAIT	The allocator can sleep.
__GFP_HIGH	The allocator can access emergency pools.
__GFP_IO	The allocator can start physical I/O.
__GFP_FS	The allocator can start filesystem I/O.
__GFP_COLD	The allocator should use cache cold pages.
__GFP_NOWARN	The allocator will not print failure warnings.
__GFP_REPEAT	The allocator will repeat the allocation if it fails, but the allocation can potentially fail. This depends upon the particular VM implementation.
__GFP_NOFAIL	The allocator will indefinitely repeat the allocation. The allocation cannot fail.
__GFP_NORETRY	The allocator will never retry if the allocation fails.
__GFP_COMP	Add compound page metadata. Used internally by thehugetlb code.
__GFP_ZERO	Add compound page metadata. Used internally by thehugetlb code.
__GFP_NOMEMALLOC	Don't use emergency reserves.
__GFP_HARDWALL	Enforce hardwall cpuset memory allocs.
__GFP_THISNODE	No fallback, no policies.
__GFP_RECLAIMABLE	Page is reclaimable.
__GFP_NOTRACK	Don't track with kmemcheck.
__GFP_NO_KSWAPD
__GFP_OTHER_NODE	On behalf of other node.

Example:

ptr = kmalloc(size, __GFP_WAIT | __GFP_IO | __GFP_FS);

 

Zone Modifiers ( 選擇要從哪分配空間 預設是從ZONE_NORMAL)

__GFP_DMA	Allocate only from ZONE_DMA
__GFP_HIGHMEM	Allocate from ZONE_HIGHMEM or ZONE_NORMAL
__GFP_DMA32
__GFP_MOVABLE

 

Type Flags( 定義了action 及 zone modifier)

GFP_ATOMIC	The allocation is high priority and must not sleep. This is the flag to use in interrupt handlers, in bottom halves, while holding a spinlock, and in other situations where you cannot sleep.
GFP_NOIO	This allocation can block, but must not initiate disk I/O. This is the flag to use in block I/O code when you cannot cause more disk I/O, which might lead to some unpleasant recursion.
GFP_NOFS	This allocation can block and can initiate disk I/O, if it must, but will not initiate a filesystem operation. This is the flag to use in filesystem code when you cannot start another filesystem operation.
GFP_KERNEL	This is a normal allocation and might block. This is the flag to use in process context code when it is safe to sleep. The kernel will do whatever it has to in order to obtain the memory requested by the caller. This flag should be your first choice.
GFP_USER	This is a normal allocation and might block. This flag is used to allocate memory for user-space processes.
GFP_HIGHUSER	This is an allocation from ZONE_HIGHMEM and might block. This flag is used to allocate memory for user-space processes.
GFP_DMA	This is an allocation from ZONE_DMA. Device drivers that need DMA-able memory use this flag, usually in combination with one of the above.

以下列出各Type modifier的組合:

GFP_ATOMIC	__GFP_HIGH
GFP_NOIO	__GFP_WAIT
GFP_NOFS	(__GFP_WAIT | __GFP_IO)
GFP_KERNEL	(__GFP_WAIT | __GFP_IO | __GFP_FS)
GFP_USER	(__GFP_WAIT | __GFP_IO | __GFP_FS)
GFP_HIGHUSER	(__GFP_WAIT | __GFP_IO | __GFP_FS | __GFP_HIGHMEM)
GFP_DMA	__GFP_DMA

建議的用法:

Situation	Solution
Process context, can sleep	Use GFP_KERNEL
Process context, cannot sleep	Use GFP_ATOMIC, or perform your allocations with GFP_KERNEL at an earlier or later point when you can sleep
Interrupt handler	Use GFP_ATOMIC
Softirq	Use GFP_ATOMIC
Tasklet	Use GFP_ATOMIC
Need DMA-able memory, can sleep	Use (GFP_DMA | GFP_KERNEL)
Need DMA-able memory, cannot sleep	Use (GFP_DMA | GFP_ATOMIC), or perform your allocation at an earlier point when you can sleep

Example:

char *reloc_lp0;
reloc_lp0 = kmalloc(size,GFP_KERNEL);
WARN_ON(!reloc_lp0);
if (!reloc_lp0) {
    pr_err("%s: Failed to allocate reloc_lp0\n",__func__);
   goto out;
}
...
kfree(reloc_lp0);

Reference:
include/linux/gfp.h
http://www.makelinux.net/books/lkd2/ch11lev1sec4
http://www.google.com.tw/url?sa=t&rct=j&q=&esrc=s&source=web&cd=5&ved=0CF8QFjAE&url=http%3A%2F%2Fant.comm.ccu.edu.tw%2Fcourse%2F96_Driver%2F0_Lectures%2FChap7-Getting%2520Hold%2520of%2520Memory_milk.ppt&ei=TMFqT_ysHqWfmQW5obyWBg&usg=AFQjCNGvVEk9-Io481khM_SqoFd-WsvwqQ&sig2=qvOIAsidZzvw15vpq0JlrQ

ObjectOutputStream送出的物件指向舊的物件?

假設我在Server端有以下的Code

HashMap<String,String> map=new HashMap<String,String> ();
ObjectOutputStream oos=new ObjectOutputStream(clientsocket.getOutputStream);
while(true){
       map.put(Long.toString(System.currentTimeMillis()),Long.toString(System.currentTimeMillis()));
       oos.writeObject(map);
}

另外一端收到的會發現後面收到的物件都跟第一次收到的物件一樣，Why?因為在送出的時候都是指向同一個位址，因此必須要加上oos.reset();結果才會正確。

File Transfer via Network using C vs. JAVA

這次的測試分別比較了使用gcc /gcc -O2 編譯而成的與javac 1.6.0_23 的版本編譯而成的執行檔來做比較，測試的兩台機器是在Lan環境下。測試的設備與程式碼如下:

CPU	Intel(R) Core(TM) i7 CPU         950  @ 3.07GHz
MEMERY	6G
NETWORK	1G
TEST FILE	466819755 byte (tar.bz2 file)

測試程式如下:
C (copy from http://content.edu.tw/primary/info_edu/cy_sa/report/more/8611a8.htm)
server.c

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <stdio.h>
#include <string.h>
#define TRUE 1
main()
{
int sock, msgsock, length;
struct sockaddr_in name;
FILE *fp;
int rval,wrval;
char buf[1024],filename[1024];

sock = socket(AF_INET, SOCK_STREAM, 0);
if (sock < 0) {
perror("opening stream socket");
exit(1);
}
/*-------------------------------------------------------*/
/* Create the acceptable connection host with Wildcard */
/*-------------------------------------------------------*/
name.sin_family = AF_INET;
name.sin_addr.s_addr = INADDR_ANY;
name.sin_port = 5000;
if (bind(sock, (struct sockaddr *)&name, sizeof name) < 0) {
perror ("binding stream socket");
exit(1);
}
listen(sock,5);
do{
msgsock = accept(sock,(struct sockaddr *)0, (int *)0);
if (msgsock == -1)
perror("accept error !\\n ");
else do {
bzero(buf, sizeof(buf) );
/*--------------------------------------------------*/
/* Read filename from msgsock (sent from client) */
/*--------------------------------------------------*/
if ((rval = read(msgsock, buf,1024)) < 0)
perror("reading stream message");
if (rval == 0)
printf("Ending connection\\n");
else {
if ((fp=fopen(buf,"r")) == NULL) {
printf("file not found : %s !\\n ",buf);
break;
}
printf("opening [%s] !!\\n ",buf);
bzero(filename,sizeof(filename));
strcpy(filename, buf);
/*--------------------------------------------------*/
/* Read file content & sent to msgsock ...> client */
/*--------------------------------------------------*/
while (fgets(buf,sizeof(buf),fp) != (char*)NULL) {
if ((wrval=write(msgsock,buf,sizeof(buf))) <0)
break;
//printf(".");
}
fclose(fp);
printf("\\n\\n closing [%s] ! \\n",filename);
rval=0;
}
}while (rval != 0);
close(msgsock);
printf(" [%s]finish* !! \\n ",filename);
}while (TRUE);
close(sock);
exit(0);
}

client.c

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <netdb.h>
#include <stdio.h>
#define DATA "The sea is clam ...."
main(int argc, char *argv[] )
{
int sock, rval, wrval;
char buf[1024];
struct sockaddr_in server;
struct hostent *hp, *gethostbyname();
FILE *fp;

sock = socket(AF_INET,SOCK_STREAM,0);
if (sock < 0) {
perror("opening stream socket");
exit(1);
}
server.sin_family=AF_INET;
hp=gethostbyname(argv[1]);
if (hp == 0 ) {
printf( "%s : unknown host\\n", argv[1]);
exit(2);
}
bcopy((char*)hp->h_addr, (char *)&server.sin_addr, hp->h_length);
server.sin_port = 5000;
if (connect(sock, (struct sockaddr *)&server, sizeof server) < 0)
{
perror ("connecting stream socket");
exit(1);
}

if (write(sock, argv[3],strlen(argv[3])) < 0)
perror ("writing stream socket");
sleep(1);
printf("requiring [%s], ...\\n",argv[3]);
fp=fopen(argv[4],"w+");
/*---------------------------------------------------------*/
/* Read file content from sock (sent from ftp_serv ) */
/* and Write to local file */
/*---------------------------------------------------------*/
while ( (rval=read(sock,buf,sizeof(buf) )) >1) {
//printf(".");
fputs(buf,fp);
}
fclose(fp);
close(sock);
printf("finish !! \\n");
printf("closing connection !! \\n");
exit(0);
}

JAVA
server.java

import java.io.BufferedInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.ServerSocket;
import java.net.Socket;

public class server {
  public static void main(String[] args) throws IOException {
    ServerSocket servsock = new ServerSocket(5000);
    File myFile = new File("hadoop.tar.bz2");
    while (true) {
      Socket sock = servsock.accept();
      byte[] mybytearray = new byte[(int) myFile.length()];
      BufferedInputStream bis = new BufferedInputStream(new FileInputStream(myFile));
      bis.read(mybytearray, 0, mybytearray.length);
      OutputStream os = sock.getOutputStream();
      os.write(mybytearray, 0, mybytearray.length);
      os.flush();
      sock.close();
    }
  }
}

client.java

import java.io.BufferedOutputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.net.Socket;

public class client {
  public static void main(String[] argv) throws Exception {
    Socket sock = new Socket("192.168.254.1", 5000);
    byte[] mybytearray = new byte[1024];
    InputStream is = sock.getInputStream();
    FileOutputStream fos = new FileOutputStream("hadoop.tar.bz2");
    BufferedOutputStream bos = new BufferedOutputStream(fos);
    int bytesRead ;
   while((bytesRead= is.read(mybytearray, 0, mybytearray.length))!=-1){
    bos.write(mybytearray, 0, bytesRead);
    }
    bos.close();
    sock.close();
  }
}

測試結果如下:

Test Case	Time(s)
C with -O2	20.539(s)
C without -O2	20.286(s)
JAVA	4.923(s)

很詭異的結果，再研究看看。

Java的List, Set, Map比較

首先我主要比較了
List: ArrayList, LinkedList
Set: HashSet, TreeSet
Map: HashMap, TreeMap
並且針對了這幾個新增刪除以及取得(Get)來做比較，測試方式是先新增10000個字串物件，然後刪除字串為"5000”的物件，再來則是利用while loop及foreach來展開全部的物件。
實驗環境為

CPU	Intel core i7 860 @2.80GHz
OS	WIndows 7 X86_64
MEM	6G
JDK	Java SE 6 Update 23

以下為數據結果:

	add(push)	delete(remove)	Get(foreach)	Get (while)
ArrayList	10	1	2	4
LinkedList	4	1	1	4
HashSet	13	0	2	5
TreeSet	38	0	4	6
HashMap	3	0	N/A	4
TreeMap	7	0	N/A	4

程式碼如下

import java.util.ArrayList;
import java.util.Collection;
import java.util.HashMap;
import java.util.HashSet;
import java.util.Iterator;
import java.util.LinkedList;
import java.util.TreeMap;
import java.util.TreeSet;


public class PerformanceTest {
 
 private static void ArrayListTest(int size){
  System.out.println("\nArrayListTest, Total size: "+size);
  ArrayList<String> list=new ArrayList<String>();
  long start=System.currentTimeMillis();
  for(int i=0;i<size;i++){
   list.add(Integer.toString(i));
  }
  System.out.println("1.Add time: "+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  
   list.remove(Integer.toString(size/2));
  
  System.out.println("2.Delete time(delete the object "+size/2+") :"+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  Iterator iterator = list.iterator();
  String s;
        while(iterator.hasNext()) {
         s=(String) iterator.next();
        }
  System.out.println("3.Get time(use iterator): "+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  for(String i : list) {
   s=i;
        }
  System.out.println("4.Get time(use enhanced for loop): "+(System.currentTimeMillis()-start)+" (ms)");
 }
 
 private static void LinkedListTest(int size){
  System.out.println("\nLinkedListTest, Total size: "+size);
  LinkedList<String> list=new LinkedList<String>();
  long start=System.currentTimeMillis();
  for(int i=0;i<size;i++){
   list.add(Integer.toString(i));
  }
  System.out.println("1.Add time: "+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  
   list.remove(Integer.toString(size/2));
  
  System.out.println("2.Delete time(delete the object "+size/2+") :"+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  Iterator iterator = list.iterator();
  String s;
        while(iterator.hasNext()) {
         s=(String) iterator.next();
        }
  System.out.println("3.Get time(use iterator): "+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  for(String i : list) {
   s=i;
        }
  System.out.println("4.Get time(use enhanced for loop): "+(System.currentTimeMillis()-start)+" (ms)");
 }
 
 private static void HashSetTest(int size){
  System.out.println("\nHashSetTest, Total size: "+size);
  HashSet<String> list=new HashSet<String>();
  long start=System.currentTimeMillis();
  for(int i=0;i<size;i++){
   list.add(Integer.toString(i));
  }
  System.out.println("1.Add time: "+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  
   list.remove(Integer.toString(size/2));
  
   System.out.println("2.Delete time(delete the object "+size/2+") :"+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  Iterator iterator = list.iterator();
  String s;
        while(iterator.hasNext()) {
         s=(String) iterator.next();
        }
  System.out.println("3.Get time(use iterator): "+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  for(String i : list) {
   s=i;
        }
  System.out.println("4.Get time(use enhanced for loop): "+(System.currentTimeMillis()-start)+" (ms)");
 }
 
 private static void TreeSetTest(int size){
  System.out.println("\nTreeSetTest, Total size: "+size);
  TreeSet<String> list=new TreeSet<String>();
  long start=System.currentTimeMillis();
  for(int i=0;i<size;i++){
   list.add(Integer.toString(i));
  }
  System.out.println("1.Add time: "+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  
   list.remove(Integer.toString(size/2));
  
   System.out.println("2.Delete time(delete the object "+size/2+") :"+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  Iterator iterator = list.iterator();
  String s;
        while(iterator.hasNext()) {
         s=(String) iterator.next();
        }
  System.out.println("3.Get time(use iterator): "+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  for(String i : list) {
   s=i;
        }
  System.out.println("4.Get time(use enhanced for loop): "+(System.currentTimeMillis()-start)+" (ms)");
 }
 
 private static void HashMapTest(int size){
  System.out.println("\nHashMapTest, Total size: "+size);
  HashMap<String,String> list=new HashMap<String,String>();
  long start=System.currentTimeMillis();
  for(int i=0;i<size;i++){
   list.put(Integer.toString(i),Integer.toString(i));
  }
  System.out.println("1.Add time: "+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  
   list.remove(Integer.toString(size/2));
  
   System.out.println("2.Delete time(delete the object "+size/2+") :"+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  Collection collection = list.values();
        Iterator iterator = collection.iterator();
        String s;
        while(iterator.hasNext()) {
            s=(String) iterator.next();
        }
  System.out.println("3.Get time(use iterator): "+(System.currentTimeMillis()-start)+" (ms)");
  
  
 }
 
 private static void TreeMapTest(int size){
  System.out.println("\nTreeMapTest, Total size: "+size);
  TreeMap<String,String> list=new TreeMap<String,String>();
  long start=System.currentTimeMillis();
  for(int i=0;i<size;i++){
   list.put(Integer.toString(i),Integer.toString(i));
  }
  System.out.println("1.Add time: "+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  
   list.remove(Integer.toString(size/2));
  
   System.out.println("2.Delete time(delete the object "+size/2+") :"+(System.currentTimeMillis()-start)+" (ms)");
  
  start=System.currentTimeMillis();
  Collection collection = list.values();
        Iterator iterator = collection.iterator();
        String s;
        while(iterator.hasNext()) {
            s=(String) iterator.next();
        }
  System.out.println("3.Get time(use iterator): "+(System.currentTimeMillis()-start)+" (ms)");
  
  
 }
 public static void main(String[] args) {
  ArrayListTest(10000);
  LinkedListTest(10000);
  HashSetTest(10000);
  TreeSetTest(10000);
  HashMapTest(10000);
  TreeMapTest(10000);
 }
}

HDFS Performance Test

很久很久以前就想測測看了！測試的項目主要有:
1.使用fuse掛載HDFS然後測試使用cp上傳下載以及使用dd產生檔案的速度
2.使用command (hadoop dfs –get/-put )取得效能
過幾天再補一個使用HDFS API來寫檔的速度好了。
以下是測試結果:

我的結論是:1.在小檔案方面基本上可以以忽略的，誤差很大(因為檔案太小)
                    2.下載比上傳快(差異蠻大的)
                    3.使用fuse掛載會慢一點點
                    4..另外參考一個數值:兩台電腦使用scp傳輸1GB時間是23秒(43.5MB/s)

Ubuntu Software RAID 5

之前拿四顆硬碟分別是 Seagate的2T*1  1.5T*3，簡單測一下效能:
A.對照組:在其中一個1.5T的硬碟切一個266G的磁區格式化為EXT4
B.實驗組:把剩下的空間作成RAID5 所以有5.7T也是格式化為EXT4
實驗方式是以dd的指令來產生檔案，bs為512K，連續跑四次扣除最快和最慢的取平均。
         1K B          512KB              1MB            10MB             100MB            1GB              10GB
A   48.9MB/s    700.5MB/s      822.5/MB/s    1091MB/s      1700MB/s       45MB/s          46MB/s
B  22.2 MB/s     553 MB/s        787MB/s       1331MB/s      1700MB/s      363MB/s        120MB/s

簡單的結論: Softraid 5 在檔案超過10MB後寫入效能會比單顆硬碟好。