Geekly articles weekly

Memilih pengarsip untuk log cadangan

Halo semuanya!


Pada artikel ini saya ingin berbicara tentang bagaimana saya memilih pengarsip untuk mengompresi log dari sistem kantor depan kami.


Divisi tempat saya bekerja terlibat dalam pengembangan dan pemeliharaan sistem front office Bank yang terpadu. Saya bertanggung jawab atas pemeliharaan, pemantauan, dan pengembangannya.


Sistem kami adalah aplikasi yang sangat banyak melayani lebih dari 5.000 pengguna unik setiap hari. Hari ini adalah "monolit" dengan semua kelebihan dan kekurangannya. Tetapi sekarang proses transfer fungsionalitas ke layanan microser sedang aktif.


Setiap hari, sistem kami menghasilkan lebih dari 130 GB log mentah, dan terlepas dari kenyataan bahwa kami menggunakan tumpukan ENG (Elasticsearch Nxlog Graylog), file log berisi lebih banyak informasi (misalnya, kesalahan pelacakan tumpukan), dan karenanya memerlukan pengarsipan dan penyimpanan.


Karena lokasi penyimpanan terbatas, muncul pertanyaan: "Dan pengarsip mana yang paling baik mengatasi tugas ini."


Untuk mengatasi masalah ini, saya menulis skrip PowerShell yang melakukan analisis untuk saya.


Tugas skrip adalah memanggil rar, 7z, dan pengarsip zip dengan parameter kompresi yang berbeda, untuk menghitung kecepatan pembentukan arsip, serta ruang disk yang digunakan.


ArchSearch.ps1
#Requires -Version 4.0 #  ,      Clear-Host #   ,     Set-Location $PSScriptRoot #  ,      $Archive = "Archive" $ArchFileName = "ArchFileName" [array]$path = (Get-ChildItem '.\logs').DirectoryName|Select-Object -Unique #      -.  ,     . if ((Test-Path -Path ".\$Archive") -ne $true){ New-Item -Path .\ -Name $Archive -ItemType Directory -Force } else { #    -    Get-ChildItem .\$Archive|Remove-Item -Recurse -Force } #      [array]$table=@() #   Rar #rar # m<0..5> Set compression level (0-store...3-default...5-maximal) 1..5|foreach{ $CompressionLevel = $("-m" + $_) $mc = Measure-Command {cmd /c .\rar.exe a -ep1 -ed $CompressionLevel -o+ -tsc .\$Archive\$($ArchFileName + $_) "$path"} [math]::Round(($mc.TotalMilliseconds), 0) $ArchFileNamePath = ".\$Archive\$($ArchFileName + $_ + ".rar")" $table += ""|Select-Object -Property @{name="ArchFileName"; expression={$ArchFileNamePath -split "\\"|Select-Object -Last 1}},` @{name="CompressionLevel"; expression={$CompressionLevel}},@{name="Extension"; expression={"rar"}},@{name="Size"; expression={(Get-ChildItem $ArchFileNamePath).Length}},` @{name="Time"; expression={[math]::Round(($mc.TotalMilliseconds), 0)}},` @{name="Size %"; expression={0}},@{name="Time %"; expression={0}},@{name="Result %"; expression={0}} } #   7z #7z # -mx[N] : set compression level: -mx1 (fastest) ... -mx9 (ultra) #cmd /c "$env:ProgramFiles\7-Zip\7z.exe" a -mx="$MX" -mmt="$MMT" -t7z -ssw -spf $($ArchFileName + "Fastest") "$path" 1..9|foreach{ $CompressionLevel = $("-mx=" + $_) $mc = Measure-Command {cmd /c "$env:ProgramFiles\7-Zip\7z.exe" a $CompressionLevel -t7z -ssw -spf .\$Archive\$($ArchFileName + $_) $path} #-mmt="$MMT" [math]::Round(($mc.TotalMilliseconds), 0) $ArchFileNamePath = ".\$Archive\$($ArchFileName + $_ + ".7z")" $table += ""|Select-Object -Property @{name="ArchFileName"; expression={$ArchFileNamePath -split "\\"|Select-Object -Last 1}},` @{name="CompressionLevel"; expression={$CompressionLevel}},@{name="Extension"; expression={"7z"}},@{name="Size"; expression={(Get-ChildItem $ArchFileNamePath).Length}},` @{name="Time"; expression={[math]::Round(($mc.TotalMilliseconds), 0)}},` @{name="Size %"; expression={0}},@{name="Time %"; expression={0}},@{name="Result %"; expression={0}} } #   zip (  PS "Compress-Archive") #zip 1..2|foreach{ Switch ($_){ 1{$CompressionLevel = "Fastest"} 2{$CompressionLevel = "Optimal"} } $mc = Measure-Command {Compress-Archive -Path $path -DestinationPath .\$Archive\$($ArchFileName + $_) -CompressionLevel $CompressionLevel -Force} [math]::Round(($mc.TotalMilliseconds), 0) $ArchFileNamePath = ".\$Archive\$($ArchFileName + $_ + ".zip")" $table += ""|Select-Object -Property @{name="ArchFileName"; expression={$ArchFileNamePath -split "\\"|Select-Object -Last 1}},` @{name="CompressionLevel"; expression={$CompressionLevel}},@{name="Extension"; expression={"zip"}},@{name="Size"; expression={(Get-ChildItem $ArchFileNamePath).Length}},` @{name="Time"; expression={[math]::Round(($mc.TotalMilliseconds), 0)}},` @{name="Size %"; expression={0}},@{name="Time %"; expression={0}},@{name="Result %"; expression={0}} } #     Size     [0] -     $Size = ($table|Sort-Object -Property Size)[0].Size / 100 #     Time     [0] -      $Time = ($table|Sort-Object -Property Time)[0].Time / 100 #    $table|foreach { $_.time $_."Size %" = [math]::Round(($_.Size / $Size), 0) $_."Time %" = [math]::Round(($_.Time / $Time), 0) if ($_."Size %" -ge $_."Time %"){ $_."Result %" = $_."Size %" - $_."Time %" } else { $_."Result %" = $_."Time %" - $_."Size %" } } #        "Size %" -     $table|Sort-Object -Property "Size %","Result %"|Select-Object -First 1|Format-Table -AutoSize #        "Result %" -          $table|Sort-Object -Property "Result %","Size %","Time %"|Select-Object -First 1|Format-Table -AutoSize $table|Sort-Object -Property "Size %","Result %"|Select-Object -First 1|Format-Table -AutoSize $table|Sort-Object -Property "Result %","Size %","Time %"|Select-Object -First 1|Format-Table -AutoSize #  !   ,     Get-ChildItem .\$Archive|Remove-Item -Force

Pekerjaan persiapan:


 #   ,   PowerShell  4.0 ($PSVersionTable): #Requires -Version 4.0 #  ,      Clear-Host #   ,     (   .\) Set-Location $PSScriptRoot 

 #  ,      $Archive = "Archive" $ArchFileName = "ArchFileName" [array]$path = (Get-ChildItem '.\logs').DirectoryName|Select-Object -Unique #      -.  ,      if ((Test-Path -Path ".\$Archive") -ne $true){ New-Item -Path .\ -Name $Archive -ItemType Directory -Force } else { #   ,     Get-ChildItem .\$Archive|Remove-Item -Recurse -Force } #      [array]$table=@()

Kami menganalisis lebih detail


 #     1  5       foreach 1..5|foreach{ 

 #    $CompressionLevel   1  5($_),       $CompressionLevel = $("-m" + $_) 

 #  Measure-Command     ,   {} $mc = Measure-Command {cmd /c .\rar.exe a -ep1 -ed $CompressionLevel -o+ -tsc .\$Archive\$($ArchFileName + $_) "$path"} 

 #       [math]::Round(($mc.TotalMilliseconds), 0) #    $ArchFileName + $_ + ".rar": Archive1.rar $ArchFileNamePath = ".\$Archive\$($ArchFileName + $_ + ".rar")" #   $table (+=)   $table += ""|Select-Object -Property @{name="ArchFileName"; expression={$ArchFileNamePath -split "\\"|Select-Object -Last 1}},` @{name="CompressionLevel"; expression={$CompressionLevel}},@{name="Extension"; expression={"rar"}},@{name="Size"; expression={(Get-ChildItem $ArchFileNamePath).Length}},` @{name="SizeAVD"; expression={0}},@{name="Time"; expression={[math]::Round(($mc.TotalMilliseconds), 0)}},` @{name="Size %"; expression={0}},@{name="Time %"; expression={0}},@{name="Result %"; expression={0}} } 

 #     Size     [0] -     $Size = ($table|Sort-Object -Property Size)[0].Size / 100 #     Time     [0] -      $Time = ($table|Sort-Object -Property Time)[0].Time / 100 #    $table|foreach { $_.time $_."Size %" = [math]::Round(($_.Size / $Size), 0) $_."Time %" = [math]::Round(($_.Time / $Time), 0) if ($_."Size %" -ge $_."Time %"){ $_."Result %" = $_."Size %" - $_."Time %" } else { $_."Result %" = $_."Time %" - $_."Size %" } } #        "Size %" -     $table|Sort-Object -Property "Size %","Result %"|Select-Object -First 1|Format-Table -AutoSize #        "Result %" -          $table|Sort-Object -Property "Result %","Size %","Time %"|Select-Object -First 1|Format-Table -AutoSize #  !   ,    . Get-ChildItem .\$Archive|Remove-Item -Force 

 $table|Sort-Object -Property "Size %","Result %"|Format-Table -AutoSize

Nama file utamaTingkat kompresiEkstensiUkuranWaktuUkuran%Waktu%Hasil%
ArchFileName8.7z-mx = 87z265115206240410016741574
ArchFileName9.7z-mx = 97z265115206461410017331633
ArchFileName7.7z-mx = 77z289481765283210914171308
ArchFileName6.7z-mx = 67z30051742373391131002889
ArchFileName5.7z-mx = 57z3123935535169118943825
ArchFileName4.rar-m4rar3351469311426126306180
ArchFileName5.rar-m5rar3346515212894126346220
ArchFileName3.rar-m3rar336980799835127264137
ArchFileName2.rar-m2rar34399885835213022494
ArchFileName4.7z-mx = 47z38926348647014717427
ArchFileName3.7z-mx = 37z44545819588916815810
ArchFileName2.7z-mx = 27z51690114475419512867
ArchFileName1.rar-m1rar53605833460020212379
ArchFileName1.7z-mx = 17z574721723728217100117
ArchFileName2.zipOptimalzip6573324214025248376128
ArchFileName1.zipTercepatzip81556824903130824266

 $table|Sort-Object -Property "Result %","Size %","Time %"|Format-Table -AutoSize

Nama file utamaTingkat kompresiEkstensiUkuranWaktuUkuran%Waktu%Hasil%
ArchFileName3.7z-mx = 37z44545819588916815810
ArchFileName4.7z-mx = 47z38926348647014717427
ArchFileName1.zipTercepatzip81556824903130824266
ArchFileName2.7z-mx = 27z51690114475419512867
ArchFileName1.rar-m1rar53605833460020212379
ArchFileName2.rar-m2rar34399885835213022494
ArchFileName1.7z-mx = 17z574721723728217100117
ArchFileName2.zipOptimalzip6573324214025248376128
ArchFileName3.rar-m3rar336980799835127264137
ArchFileName4.rar-m4rar3351469311426126306180
ArchFileName5.rar-m5rar3346515212894126346220
ArchFileName5.7z-mx = 57z3123935535169118943825
ArchFileName6.7z-mx = 67z30051742373391131002889
ArchFileName7.7z-mx = 77z289481765283210914171308
ArchFileName8.7z-mx = 87z265115206240410016741574
ArchFileName9.7z-mx = 97z265115206461410017331633

 #  !   ,    . Get-ChildItem .\$Archive|Remove-Item -Force

Hasilnya:
Yang paling ekonomis dalam hal ruang disk adalah 7z dengan rasio kompresi -mx = 8 dan -mx = 9



Waktu pembuatan arsip tercepat adalah 7z dengan rasio kompresi -mx = 1



Kecepatan dan ruang terisi yang paling optimal adalah 7z dengan rasio kompresi -mx = 3



Kami memilih 7z dengan rasio kompresi -mx = 8, karena ukuran arsip -mx 9, tetapi bekerja sedikit lebih cepat.


Hebat, pengarsipan dan rasio kompresi dipilih, sekarang mari kita arsipkan log!


Kita perlu memecahkan masalah-masalah berikut:


  1. Hindari beban server yang tinggi selama operasi.
  2. Memproses semua folder dengan subfolder Log.
  3. Hapus arsip yang lebih dari 30 hari (agar tidak kehabisan ruang disk).
  4. Buat arsip berdasarkan hari tergantung pada waktu file dimodifikasi.

ArchLogs.ps1
 #Requires -Version 4.0 #          #SCHTASKS /Create /SC DAILY /ST 22:00 /TN ArchLogs /TR "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe C:\Scripts\ArchLogs.ps1" /RU "NT AUTHORITY\NETWORKSERVICE" /F /RL HIGHEST #  ,      Clear-Host #   ,     Set-Location $PSScriptRoot #           . #  1 Function Set-MMTCore { $Core = (Get-WmiObject –class Win32_processor|Measure-Object NumberOfLogicalProcessors -sum).Sum / 2 $CoreTMP = $Core / 4 IF ($Core -lt 4) { $Core = $Core -1 } Else { $Core = $Core - $CoreTMP } [math]::Round($Core) } #    $MX = 8 #     $MMT = Set-MMTCore #  ,      $SearchFolder = "C:\inetpub" #     $SearchFolder $InetpubFolder = Get-ChildItem $SearchFolder #     Logs #  2 $LogsFolder = $InetpubFolder|foreach {Get-ChildItem $_.Fullname|Where-Object{$_.PSIsContainer -eq $true}|Where-Object{$_.name -eq "logs"}} #  ,      $Archive = "Archive" $ArchiveFile = "ArchFiles.txt" #      $LogsFolder   $LogsFolderName   foreach($LogsFolderName in $LogsFolder.Fullname){ $LogsFolderName #   $Archive,       IF ((Test-Path -Path "$LogsFolderName\$Archive") -ne $true){New-Item -Path $LogsFolderName -Name $Archive -ItemType Directory -Force} #    30  #  3 Get-ChildItem "$LogsFolderName\$Archive"|Where-Object {$_.LastWriteTime -le (Get-Date).AddDays(-30)}| Remove-Item -Force #     [Array]$ArchiveItems = Get-ChildItem -Path $LogsFolderName -Exclude $Archive #   , .    -  IF ($ArchiveItems.Count -ne ^_^quot&#0;quot^_^){ #       $AllLogsFiles = Get-ChildItem $ArchiveItems -Recurse <#-Filter *.log#>|Where-Object {$_.LastWriteTime -lt (Get-Date).Date} #     #  4 $AllData = ($AllLogsFiles|Sort-Object -Property LastWriteTime).LastWriteTime.Date|Select-Object -Unique $AllData foreach ($Data in $AllData){ IF ($ArchiveItems.Count -ne ^_^quot&#0;quot^_^){ #         ($AllLogsFiles|Where-Object {$_.LastWriteTime -lt (Get-Date).Date}|Where-Object {$_.LastWriteTime -ge (Get-Date $Data) -and $_.LastWriteTime -lt (Get-Date $Data).AddDays(1)}).FullName|Out-File .\$ArchiveFile -Force -Encoding default Write-Host "===" $Data Write-Host "===" #      IF ($(Get-Content .\ArchFiles.txt -Encoding default) -ne $null){ $ArchiveFileName =$(($LogsFolderName.Remove($LogsFolderName.LastIndexOf("\"))) -split "\\"|Select-Object -Last 1) + "_" + $(Get-Date $Data -Format dd-MM-yyyy) cmd /c "$env:ProgramFiles\7-Zip\7z.exe" a -mx="$MX" -mmt="$MMT" -t7z -ssw -sdel "$LogsFolderName\$Archive\$ArchiveFileName" "@$ArchiveFile" #        IF(Test-Path ".\$ArchiveFile"){Remove-Item ".\$ArchiveFile" -Force} #      Logs $LogsFolderName|foreach {Get-ChildItem $_|Where-Object {(Get-ChildItem -Path $_.FullName) -eq $null}|Remove-Item -Force} } } } } }

Di sini saya tidak akan melukis setiap baris skrip (dijelaskan dalam komentar), tetapi fokus hanya pada fungsi Set-MMTCore, yang memungkinkan kita menghitung jumlah utas untuk 7z agar tidak memuat prosesor pada server kami:


 Function Set-MMTCore { #        2 $Core = (Get-WmiObject –class Win32_processor|Measure-Object NumberOfLogicalProcessors -sum).Sum / 2 #      4 $CoreTMP = $Core / 4 IF ($Core -lt 4) { $Core = $Core -1 } Else { $Core = $Core - $CoreTMP } #    [math]::Round($Core) }

Tanpa menggunakan fungsi Set-MMTCore

Dapat dilihat bahwa CPU bersandar pada 100%. Ini berarti bahwa kita pasti akan menyebabkan masalah di server, serta menerima peringatan dari sistem pemantauan.



Saat menggunakan fungsi Set-MMTCore

Dapat dilihat bahwa CPU adalah 30-35%. Ini berarti menggunakan fungsi Set-MMTCore memungkinkan Anda untuk mengarsipkan file tanpa mempengaruhi operasi server.



Hasil skrip cadangan:


Folder sebelum pengarsipan:


Folder setelah pengarsipan:


File yang dibuat selama proses pengarsipan:


File dalam arsip:


Source: https://habr.com/ru/post/id484414/

More articles:


All Articles