Saya dari Luxoft.
Menurut Opennet : LeoFS adalah fasilitas penyimpanan yang toleran terhadap kesalahan untuk objek LeoFS , kompatibel dengan klien yang menggunakan Amazon S3 API dan REST-API, dan juga mendukung mode operasi sebagai server NFS. Ada optimasi untuk menyimpan objek kecil dan sangat besar, ada mekanisme caching bawaan, replikasi penyimpanan di antara pusat data dimungkinkan. Di antara tujuan proyek adalah pencapaian keandalan 99,9999999% karena replikasi duplikat yang berlebihan dan penghapusan satu titik kegagalan. Kode proyek ditulis dalam bahasa Erlang.
LeoFS terdiri dari tiga komponen:
- LeoFS Storage - layanan operasi penambahan, pengambilan dan penghapusan objek dan metadata, bertanggung jawab untuk melakukan replikasi, pemulihan, dan antrian permintaan klien.
- LeoFS Gateway - melayani permintaan HTTP dan mengalihkan tanggapan ke klien menggunakan REST-API atau S3-API, menyediakan cache data yang paling banyak diminta dalam memori dan pada disk.
- LeoFS Manager - memantau operasi LeoFS Gateway dan LeoFS Storage node, memantau status node dan memeriksa checksum. Memastikan integritas data dan ketersediaan penyimpanan yang tinggi.
Dalam posting ini, instal Leofs menggunakan ansible-playbook, test S3, NFS.
Jika Anda mencoba menginstal LeoFS menggunakan buku pedoman resmi, maka berbagai kesalahan menunggu Anda: 1 , 2 . Dalam posting ini saya akan menulis apa yang perlu dilakukan untuk menghindari kesalahan ini.
Di mana Anda akan menjalankan playbook yang memungkinkan, Anda harus menginstal netcat.
Contoh inventaris
Contoh inventaris (di repositori hosts.sample):# Please check roles/common/vars/leofs_releases for available versions [all:vars] leofs_version=1.4.3 build_temp_path="/tmp/leofs_builder" build_install_path="/tmp/" build_branch="master" source="package" #[builder] #172.26.9.177 # nodename of leo_manager_0 and leo_manager_1 are set at group_vars/all [leo_manager_0] 172.26.9.176 # nodename of leo_manager_0 and leo_manager_1 are set at group_vars/all [leo_manager_1] 172.26.9.178 [leo_storage] 172.26.9.179 leofs_module_nodename=S0@172.26.9.179 172.26.9.181 leofs_module_nodename=S0@172.26.9.181 172.26.9.182 leofs_module_nodename=S0@172.26.9.182 172.26.9.183 leofs_module_nodename=S0@172.26.9.183 [leo_gateway] 172.26.9.180 leofs_module_nodename=G0@172.26.9.180 172.26.9.184 leofs_module_nodename=G0@172.26.9.184 [leofs_nodes:children] leo_manager_0 leo_manager_1 leo_gateway leo_storage
Persiapan Server
Menonaktifkan Selinux. Saya harap komunitas membuat kebijakan Selinux untuk LeoFS.
- name: Install libselinux as prerequisite for SELinux Ansible module yum: name: "{{item}}" state: latest with_items: - libselinux-python - libsemanage-python - name: Disable SELinux at next reboot selinux: state: disabled - name: Set SELinux in permissive mode until the machine is rebooted command: setenforce 0 ignore_errors: true changed_when: false
Instal netcat dan redhat-lsb-core . netcat diperlukan untuk leofs-adm , redhat-lsb-core diperlukan untuk menentukan versi OS di sini .
- name: Install Packages yum: name={{ item }} state=present with_items: - nmap-ncat - redhat-lsb-core
Membuat leof pengguna dan menambahkannya ke grup wheel
- name: Create user leofs group: name: leofs state: present - name: Allow 'wheel' group to have passwordless sudo lineinfile: dest: /etc/sudoers state: present regexp: '^%wheel' line: '%wheel ALL=(ALL) NOPASSWD: ALL' validate: 'visudo -cf %s' - name: Add the user 'leofs' to group 'wheel' user: name: leofs groups: wheel append: yes
Instal Erlang
- name: Remote erlang-20.3.8.23-1.el7.x86_64.rpm install with yum yum: name=https://github.com/rabbitmq/erlang-rpm/releases/download/v20.3.8.23/erlang-20.3.8.23-1.el7.x86_64.rpm
Versi lengkap dari buku pedoman yang dikoreksi dapat ditemukan di sini: https://github.com/patsevanton/leofs_ansible
Instalasi, konfigurasi, mulai
Selanjutnya, jalankan seperti yang tertulis di https://github.com/leo-project/leofs_ansible tanpa build_leofs.yml
Memeriksa status kluster pada LeoManager Utama
leofs-adm status
Primer dan Sekunder dapat dilihat di log playbook ansible
Kesimpulannya akan seperti ini [System Confiuration] -----------------------------------+---------- Item | Value -----------------------------------+---------- Basic/Consistency level -----------------------------------+---------- system version | 1.4.3 cluster Id | leofs_1 DC Id | dc_1 Total replicas | 2 number of successes of R | 1 number of successes of W | 1 number of successes of D | 1 number of rack-awareness replicas | 0 ring size | 2^128 -----------------------------------+---------- Multi DC replication settings -----------------------------------+---------- [mdcr] max number of joinable DCs | 2 [mdcr] total replicas per a DC | 1 [mdcr] number of successes of R | 1 [mdcr] number of successes of W | 1 [mdcr] number of successes of D | 1 -----------------------------------+---------- Manager RING hash -----------------------------------+---------- current ring-hash | a0314afb previous ring-hash | a0314afb -----------------------------------+---------- [State of Node(s)] -------+----------------------+--------------+---------+----------------+----------------+---------------------------- type | node | state | rack id | current ring | prev ring | updated at -------+----------------------+--------------+---------+----------------+----------------+---------------------------- S | S0@172.26.9.179 | running | | a0314afb | a0314afb | 2019-12-05 10:33:47 +0000 S | S0@172.26.9.181 | running | | a0314afb | a0314afb | 2019-12-05 10:33:47 +0000 S | S0@172.26.9.182 | running | | a0314afb | a0314afb | 2019-12-05 10:33:47 +0000 S | S0@172.26.9.183 | attached | | | | 2019-12-05 10:33:58 +0000 G | G0@172.26.9.180 | running | | a0314afb | a0314afb | 2019-12-05 10:33:49 +0000 G | G0@172.26.9.184 | running | | a0314afb | a0314afb | 2019-12-05 10:33:49 +0000 -------+----------------------+--------------+---------+----------------+----------------+----------------------------
Buat pengguna
Buat leof pengguna:
leofs-adm create-user leofs leofs access-key-id: 9c2615f32e81e6a1caf5 secret-access-key: 8aaaa35c1ad78a2cbfa1a6cd49ba8aaeb3ba39eb
Daftar pengguna:
leofs-adm get-users user_id | role_id | access_key_id | created_at ------------+---------+------------------------+--------------------------- _test_leofs | 9 | 05236 | 2019-12-02 06:56:49 +0000 leofs | 1 | 9c2615f32e81e6a1caf5 | 2019-12-02 10:43:29 +0000
Buat Bucket
Membuat ember
leofs-adm add-bucket leofs 9c2615f32e81e6a1caf5 OK
Daftar ember:
leofs-adm get-buckets cluster id | bucket | owner | permissions | created at -------------+----------+--------+------------------+--------------------------- leofs_1 | leofs | leofs | Me(full_control) | 2019-12-02 10:44:02 +0000
Mengkonfigurasi s3cmd
Di bidang HTTP Proxy server name , tentukan IP server Gateway
s3cmd --configure Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key [9c2615f32e81e6a1caf5]: Secret Key [8aaaa35c1ad78a2cbfa1a6cd49ba8aaeb3ba39eb]: Default Region [US]: Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3. S3 Endpoint [s3.amazonaws.com]: Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used if the target S3 system supports dns based buckets. DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: leofs Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [No]: On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name [172.26.9.180]: HTTP Proxy server port [8080]: New settings: Access Key: 9c2615f32e81e6a1caf5 Secret Key: 8aaaa35c1ad78a2cbfa1a6cd49ba8aaeb3ba39eb Default Region: US S3 Endpoint: s3.amazonaws.com DNS-style bucket+hostname:port template for accessing a bucket: leofs Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: 172.26.9.180 HTTP Proxy server port: 8080 Test access with supplied credentials? [Y/n] Y Please wait, attempting to list all buckets... Success. Your access key and secret key worked fine :-) Now verifying that encryption works... Not configured. Never mind. Save settings? [y/N] y Configuration saved to '/home/user/.s3cfg'
Jika Anda menerima galat ERROR: S3 error: 403 (AccessDenied): Access Denied:
s3cmd put test.py s3://leofs/ upload: 'test.py' -> 's3://leofs/test.py' [1 of 1] 382 of 382 100% in 0s 3.40 kB/s done ERROR: S3 error: 403 (AccessDenied): Access Denied
Maka Anda perlu memperbaiki signature_v2 di True s3cmd config. Detail dalam masalah ini.
Jika signature_v2 salah, maka akan ada kesalahan seperti itu:
WARNING: Retrying failed request: /?delimiter=%2F (getaddrinfo() argument 2 must be integer or string) WARNING: Waiting 3 sec... WARNING: Retrying failed request: /?delimiter=%2F (getaddrinfo() argument 2 must be integer or string) WARNING: Waiting 6 sec... ERROR: Test failed: Request failed for: /?delimiter=%2F
Pengujian boot
Buat file 1GB
fallocate -l 1GB 1gb
Unggah ke Leofs
time s3cmd put 1gb s3://leofs/ real 0m19.099s user 0m7.855s sys 0m1.620s
Statistik
leofs-adm du untuk 1 simpul:
leofs-adm du S0@172.26.9.179 active number of objects: 156 total number of objects: 156 active size of objects: 602954495 total size of objects: 602954495 ratio of active size: 100.0% last compaction start: ____-__-__ __:__:__ last compaction end: ____-__-__ __:__:__
Kami melihat bahwa kesimpulannya tidak terlalu informatif.
Mari kita lihat di mana file ini berada.
leofs-adm whereis leofs / 1gb
leofs-adm whereis leofs/1gb -------+----------------------+--------------------------------------+------------+--------------+----------------+----------------+----------------+---------------------------- del? | node | ring address | size | checksum | has children | total chunks | clock | when -------+----------------------+--------------------------------------+------------+--------------+----------------+----------------+----------------+---------------------------- | S0@172.26.9.181 | 657a9f3a3db822a7f1f5050925b26270 | 976563K | a4634eea55 | true | 64 | 598f2aa976a4f | 2019-12-05 10:48:15 +0000 | S0@172.26.9.182 | 657a9f3a3db822a7f1f5050925b26270 | 976563K | a4634eea55 | true | 64 | 598f2aa976a4f | 2019-12-05 10:48:15 +0000
Aktifkan NFS
Kami mengaktifkan NFS di server Leo Gateway 172.26.9.184.
Di server dan klien, instal nfs-utils
sudo yum install nfs-utils
Menurut instruksi, kita akan memperbaiki file konfigurasi /usr/local/leofs/current/leo_gateway/etc/leo_gateway.conf
protocol = nfs
Di server 172.26.9.184, jalankan rpcbind dan leofs-gateway
sudo service rpcbind start sudo service leofs-gateway restart
Di server tempat leo_manager berjalan, buat ember untuk NFS dan buat kunci untuk terhubung ke NFS
leofs-adm add-bucket test 05236 leofs-adm gen-nfs-mnt-key test 05236 ip--nfs-
Terhubung ke NFS
sudo mkdir /mnt/leofs
Lihat ruang disk melalui klien NFS
Ruang disk mengingat setiap node penyimpanan memiliki disk 40GB (3 node berjalan, 1 node terlampir):
df -hP Filesystem Size Used Avail Use% Mounted on 172.26.9.184:/test/05236/e7298032e78749149dd83a1e366afb328811c95b 60G 3.6G 57G 6% /mnt/leofs
Instal LeoFS dengan 6 node penyimpanan.
Persediaan (tanpa pembangun): Output status Leofs-adm
Output status Leofs-adm [System Confiuration] -----------------------------------+---------- Item | Value -----------------------------------+---------- Basic/Consistency level -----------------------------------+---------- system version | 1.4.3 cluster Id | leofs_1 DC Id | dc_1 Total replicas | 2 number of successes of R | 1 number of successes of W | 1 number of successes of D | 1 number of rack-awareness replicas | 0 ring size | 2^128 -----------------------------------+---------- Multi DC replication settings -----------------------------------+---------- [mdcr] max number of joinable DCs | 2 [mdcr] total replicas per a DC | 1 [mdcr] number of successes of R | 1 [mdcr] number of successes of W | 1 [mdcr] number of successes of D | 1 -----------------------------------+---------- Manager RING hash -----------------------------------+---------- current ring-hash | d8ff465e previous ring-hash | d8ff465e -----------------------------------+---------- [State of Node(s)] -------+----------------------+--------------+---------+----------------+----------------+---------------------------- type | node | state | rack id | current ring | prev ring | updated at -------+----------------------+--------------+---------+----------------+----------------+---------------------------- S | S0@172.26.9.178 | running | | d8ff465e | d8ff465e | 2019-12-06 05:18:29 +0000 S | S0@172.26.9.179 | running | | d8ff465e | d8ff465e | 2019-12-06 05:18:29 +0000 S | S0@172.26.9.181 | running | | d8ff465e | d8ff465e | 2019-12-06 05:18:30 +0000 S | S0@172.26.9.182 | running | | d8ff465e | d8ff465e | 2019-12-06 05:18:29 +0000 S | S0@172.26.9.183 | running | | d8ff465e | d8ff465e | 2019-12-06 05:18:29 +0000 S | S0@172.26.9.185 | running | | d8ff465e | d8ff465e | 2019-12-06 05:18:29 +0000 G | G0@172.26.9.180 | running | | d8ff465e | d8ff465e | 2019-12-06 05:18:31 +0000 G | G0@172.26.9.184 | running | | d8ff465e | d8ff465e | 2019-12-06 05:18:31 +0000 -------+----------------------+--------------+---------+----------------+----------------+----------------------------
Ruang disk, dengan mempertimbangkan bahwa setiap node penyimpanan memiliki disk 40GB (6 node berjalan):
df -hP Filesystem Size Used Avail Use% Mounted on 172.26.9.184:/test/05236/e7298032e78749149dd83a1e366afb328811c95b 120G 3.6G 117G 3% /mnt/leofs
Jika 5 node penyimpanan digunakan
[leo_storage] 172.26.9.178 leofs_module_nodename=S0@172.26.9.178 172.26.9.179 leofs_module_nodename=S1@172.26.9.179 172.26.9.181 leofs_module_nodename=S2@172.26.9.181 172.26.9.182 leofs_module_nodename=S3@172.26.9.182 172.26.9.183 leofs_module_nodename=S4@172.26.9.183
df -hP 172.26.9.184:/test/05236/e7298032e78749149dd83a1e366afb328811c95b 100G 3.0G 97G 3% /mnt/leofs
Log
Log berada di direktori /usr/local/leofs/current/*/log
Jika Anda menginstal / mengkonfigurasi Leofs secara manual, Anda mungkin mengalami kesalahan berikut.
[GALAT] Mnesia tidak tersedia
Mulai layanan systemctl mulai leofs-manager-master
leofs-adm status [ERROR] Mnesia is not available
Perlu memulai systemctl start leofs-manager-slave di leo_manager_1
Penyimpanan leofs tidak dimulai.
Diperlukan untuk menjalankan leofs-manager-master dan leofs-manager-slave dan leofs-adm untuk menunjukkan statusnya.
Node terlampir kurang dari # replika
Ketika Anda memulai leofs-adm mulai, Anda mendapatkan kesalahan ini:
leofs-adm start [ERROR] Attached nodes less than
Tidak cukup node Penyimpanan. status leofs-adm akan menunjukkan kepada Anda kurang dari 2 node penyimpanan. Jumlah minimum yang diperlukan dari node Penyimpanan 2.
Status leofs-adm menunjukkan terlampir, sisanya berjalan.
Perlu menyeimbangkan kembali node
leofs-adm rebalance
Setelah memulai leofs-gateway, Anda tidak melihat node Gateway dalam status leofs-adm
Perlu memulai leofs-adm
leofs-adm start
tidak dapat terhubung ke LeoFS Manager pada node Slave
(Secara default, leofs-adm tidak bekerja pada slave node!] ( Https://leo-project.net/leofs/docs/issues/documentation-issues/ )
Uji beban
Pengujian dilakukan pada 2 node dengan konfigurasi:
CPU: Single Core Intel Core (Broadwell) (-MCP-) speed: 2295 MHz Kernel: 3.10.0-862.3.2.el7.x86_64 x86_64 Up: 1h 08m Mem: 1023.8/1999.6 MiB (51.2%) Storage: 10.00 GiB (43.5% used) Procs: 98 Shell: bash 4.2.46 inxi: 3.0.37
Untuk pengujian, ambil disk kecil
Pada kedua node kita melihat disk ruang kosong 9,4G dan 5,9G.
df -hP Filesystem Size Used Avail Use% Mounted on /dev/vda1 9.4G 5.9G 3.1G 66% /
Saluran Telegram: SDS dan Cluster FS