文章更新历史：
2014-12-27 介绍ceph集群搭建,每台OSD主机跑1个OSD CRUSH2014-12-29 改进自动配置脚本，每台OSD主机跑三个OSD CRUSH
环境简介：
总共五台主机：192.168.239.131 ceph-mon192.168.239.132 ceph-mds192.168.239.160 ceph-osd0192.168.239.161 ceph-osd1192.168.239.162 ceph-osd2
期望目标：
在上述主机中跑三个mon节点，三个osd节点，两个mds节点，其中每个OSD节点跑三个OSD CRUSHmon节点：ceph-mon , ceph-mds , ceph-osd0osd节点：ceph-osd0 , ceph-osd1 , ceph-osd2mds节点： ceph-mon , ceph-mds
1、从suse.com官网注册一个账号，下载SLES 11 SP3和SUSE Cloud 4的ISO
2、给每个节点安装系统，然后设置两个安装源,一个OS，一个SUSE Cloud 4
3、配置ceph-mon到其他节点的root用户无密码登录ssh
4、复制ceph-mon节点的/etc/hosts到其他节点
5、安装ceph软件包
# zypper -n install ceph ceph-radosgw
6、在ceph-mon节点上，使用setup.sh分别调用init-mon.sh , init-osd.sh , init-mds.sh自动配置mon,osd,mds。
setup.sh和init-mon.sh会进入当前目录下的./ceph文件夹，请务必在/etc以外的目录执行。
各个脚本的代码如下(仅供参考)：
(1) setup.sh
#!/bin/bash### Stop all existed OSD nodesprintf "Killing all ceph-osd nodes..."for i in 0 1 2;do ssh ceph-osd$i "killall -TERM ceph-osd" sleep 1doneprintf "Done\n"printf "Killing all mon nodes..."ssh ceph-mon killall -TERM ceph-monprintf "Done\n"### Initialize mon on this systemprintf "Initializing ceph-mon nodes..."./init-mon.shcd ./cephprintf "Done\n"### Initialize osd services on nodesfor i in 0 1 2;do ../init-osd.sh ceph-osd$i $i sleep 1 done### Initialize mds on remote nodeprintf "Initializing mds..."for node in ceph-mon ceph-mds;do ../init-mds.sh $nodedoneprintf "Done\n"
(2) init-mon.sh
#!/bin/bashfsid=$(uuidgen)mon_node=ceph-mon,ceph-mds,ceph-osd0cluster_net=192.168.239.0/24public_net=192.168.1.0/24node_list=$(echo $mon_node | tr ',' ' ')for node in $node_list;do node_ip=$(ping -c 1 $node | head -n 1 | awk '{ print $3 }' | sed -e 's/(//' -e 's/)//') if [ -n "$mon_ip_list" ];then mon_ip_list="$mon_ip_list,$node_ip" else mon_ip_list="$node_ip" fidonecat > ./clean-mon.sh < ceph.conf << EOF[global] fsid = $fsid mon initial members = $mon_node mon host = $mon_ip_list public network = $public_net cluster network = $cluster_net auth cluster required = cephx auth service required = cephx auth client required = cephx osd journal size = 1024 filestore xattr use omap = trueEOFceph-authtool --create-keyring bootstrap-osd.keyring --gen-key -n client.bootstrap-osdceph-authtool --create-keyring bootstrap-mds.keyring --gen-key -n client.bootstrap-mdsceph-authtool --create-keyring ceph.mon.keyring --gen-key -n mon. --cap mon 'allow *'ceph-authtool --create-keyring ceph.client.admin.keyring --gen-key -n client.admin --set-uid=0 --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow'ceph-authtool ceph.mon.keyring --import-keyring ceph.client.admin.keyringport_num=6789for node in $node_list;do scp -p ceph.conf ceph.client.admin.keyring $node:/etc/ceph node_ip=$(ping -c 1 $node | head -n 1 | awk '{ print $3 }' | sed -e 's/(//' -e 's/)//') map_list="$map_list --add $node $node_ip:$port_num"donemonmaptool --create ./monmap.ceph $map_list --fsid $(grep fsid /etc/ceph/ceph.conf | awk '{ print $NF}') for node in $node_list;do scp ./monmap.ceph $node:/tmp/ ### Make filesystem for ceph-mon scp -p ceph.mon.keyring $node:/tmp ssh $node ceph-mon --mkfs -i $node --monmap /tmp/monmap.ceph --keyring /tmp/ceph.mon.keyring --mon-data /data/mon.$node ### Start the ceph-mon service ssh $node ceph-mon -i $node --mon-data /data/mon.$node printf "Waiting for mon.$node started..." sleep 5 printf "Done\n"done### Initialize bootstrap keyringsceph auth add client.bootstrap-mds mon 'allow profile bootstrap-mds' -i bootstrap-mds.keyringceph auth add client.bootstrap-osd mon 'allow profile bootstrap-osd' -i bootstrap-osd.keyring (3) init-osd.sh
#!/bin/bashif [ $# -lt 1 ];then printf "Usage:$0 {host}\n" $0 exit 1fihost=$1weight=1.0### Add the host bucket to osd treeceph osd crush add-bucket $host hostceph osd crush move $host root=default### Clean the OSD deomons firstssh $host "killall -TERM ceph-osd"ssh $host "rm -f /var/lib/ceph/bootstrap-osd/*keyring"ssh $host "mkdir -p /var/lib/ceph/bootstrap-osd"### Copy configuration files to destination hostscp ceph.conf ceph.client.admin.keyring $host:/etc/cephscp bootstrap-osd.keyring $host:/var/lib/ceph/bootstrap-osd/ceph.keyring### Every host run 3 OSD daemonsprintf "Initializing OSD for $host...\n"for i in 0 1 2;do osd_uuid=`uuidgen` osd_num=$(ssh $host "ceph osd create $osd_uuid 2>&1") ssh $host "rm -fr /data/osd.$osd_num" ssh $host "mkdir -p /data/osd.$osd_num" ssh $host "ceph-disk-prepare --osd-uuid $osd_uuid --fs-type xfs /data/osd.$osd_num" ssh $host "ceph-osd -i $osd_num --osd-data /data/osd.$osd_num --osd-journal /data/osd.$osd_num/journal --mkkey --mkfs" ssh $host "ceph auth add osd.$osd_num osd 'allow *' mon 'allow profile osd' -i /data/osd.$osd_num/keyring" ssh $host "ceph osd crush add osd.$osd_num $weight host=$host" ssh $host "ceph-osd -i $osd_num --osd-data /data/osd.$osd_num --osd-journal /data/osd.$osd_num/journal" sleep 1doneprintf "Initializing OSD for $host...Done\n" (4) init-mds.sh
#!/bin/bashif [ $# -lt 1 ];then printf "Usage:$0 {host}}\n" $0 exit 1fimds_host=$1mds_name=mds.$mds_hostmds_data=/data/$mds_namekeyfile=ceph.$mds_host.keyringmon_host=ceph-mon:6789### Stop current running mds daemons firstssh $mds_host "killall -TERM ceph-mds"ssh $mds_host "rm -fr $mds_data"ssh $mds_host "mkdir -p $mds_data"### Clean the old keyring file firstif [ ! -e $keyfile ];then ### Create new keyring file ceph-authtool -C -g -n $mds_name $keyfileficeph auth add $mds_name mon 'allow profile mds' osd 'allow rwx' mds 'allow' -i $keyfilescp -p \/etc/ceph/ceph.conf \/etc/ceph/ceph.client.admin.keyring $mds_host:/etc/cephscp -p $keyfile $mds_host:$mds_data/keyringssh $mds_host "ceph-mds -i $mds_host -n $mds_name -m $mon_host --mds-data=/data/mds.$mds_host" 脚本执行完之后会自动把服务启动，在ceph-mon节点上查看ceph集群状态：
ceph-mon:~ # ceph -s cluster 484f6e40-e047-4448-a2e4-678493d122c6 health HEALTH_OK monmap e1: 3 mons at {ceph-mds=192.168.239.132:6789/0,ceph-mon=192.168.239.131:6789/0,ceph-osd0=192.168.239.160:6789/0}, election epoch 6, quorum 0,1,2 ceph-mon,ceph-mds,ceph-osd0 mdsmap e4: 1/1/1 up {0=ceph-mon=up:active}, 1 up:standby osdmap e44: 9 osds: 9 up, 9 in pgmap v58: 192 pgs, 3 pools, 1884 bytes data, 20 objects 28136 MB used, 116 GB / 143 GB avail 192 active+clean osd状态：
ceph-mon:~ # ceph osd tree# id weight type name up/down reweight-1 9 root default-2 3 host ceph-osd00 1 osd.0 up 1 1 1 osd.1 up 1 2 1 osd.2 up 1 -3 3 host ceph-osd13 1 osd.3 up 1 4 1 osd.4 up 1 5 1 osd.5 up 1 -4 3 host ceph-osd26 1 osd.6 up 1 7 1 osd.7 up 1 8 1 osd.8 up 1
在ceph-mon节点上查看进程：
ceph-mon:~ # ps ax |grep ceph-mon 8993 pts/0 Sl 0:00 ceph-mon -i ceph-mon --mon-data /data/ceph-mon
在ceph-osdX节点上查看进程：
ceph-osd0:~ # ps ax | grep ceph-osd 13140 ? Ssl 0:02 ceph-osd -i 0 --osd-data /data/osd.0 --osd-journal /data/osd.0/journal
在ceph-mds节点上查看进程：
ceph-mds:~ # ps ax |grep ceph-mds 42260 ? Ssl 0:00 ceph-mds -i ceph-mds -n mds.ceph-mds -m ceph-mon:6789 --mds-data=/data/mds.ceph-mds
7、挂载cephfs的两种方式
(1) mount.ceph
由于SLES 11系列的内核还不支持ceph模块，您需要在客户端安装较高版本的内核才能获得mount.ceph的功能。mount.ceph命令用法如下：
mount.ceph {mon ip/host}:/ {mount point} -o name=admin,secret={your keyring}
mount.ceph ceph-mon:/ /mnt/cephfs -v -o name=admin,secret=AQD5jp5UqPRtCRAAvpRyhlNI0+qEHjZYqEZw8A==
secret指定的密钥从/etc/ceph/ceph.client.admin.keyring文件获得，出于安全考虑，为了避免讲密钥暴露在命令历史中，请使用secretfile=指定包含密钥的文件。
(2) ceph-fuse
没有内核ceph模块的情况下可以用ceph-fuse来挂载文件系统，用法如下：
ceph-fuse ceph-mon:6789 /mnt/cephfs
8、查看挂载状态：
ceph-mon:/etc/ceph # df -Ph文件系统 容量 已用 可用 已用% 挂载点/dev/mapper/rootvg-root 12G 5.3G 5.7G 49% /udev 12G 5.3G 5.7G 49% /devtmpfs 12G 5.3G 5.7G 49% /dev/shm/dev/sda1 185M 36M 141M 21% /boot/dev/sdb1 16G 35M 16G 1% /data192.168.239.131:/ 48G 3.2G 45G 7% /mnt/cephfs