lxc/lxd container 建立與管理及佈署基本指令 – part. 1

首先登入 lxc

lxc exec lxc00 bash

或是

lxc shell lxc00

安裝及設定必備的環境

sudo cp /etc/apt/sources.list sources.list.default
sudo sed -i 's/archive.ubuntu.com/free.nchc.org.tw/g' /etc/apt/sources.list
sudo apt update && sudo apt dist-upgrade
sudo apt install bash-completion net-tools command-not-found byobu openssh-server curl man-db software-properties-common byobu dnsutils ppa-purge iotop sysstat sshfs virt-what htop git unzip rsync apt-transport-https

必備環境安裝說明:

  • 變更 apt server 到離你最近或是頻寬最佳的,可以參考這篇 挑選最佳的 Ubuntu apt mirror
  • 更新 apt db
  • 安裝基本環境軟體
    • bash-completion net-tools command-not-found byobu openssh-server curl man-db software-properties-common byobu dnsutils ppa-purge iotop sysstat sshfs virt-what

設定 auto completion (ubuntu 這個帳號也要一次)

echo "source /etc/profile.d/bash_completion.sh" >> ~/.bashrc
source /etc/profile.d/bash_completion.sh
su -l ubuntu && echo "source /etc/profile.d/bash_completion.sh" >> ~/.bashrc
source /etc/profile.d/bash_completion.sh

修改密碼

sudo passwd
sudo passwd ubuntu

產生金鑰不用密碼登入

ssh-keygen -t ed25519 -f ~/.ssh/192.168.1.2 -C ubuntu@192.168.1.2
ssh-copy-id -p 22 -i ~/.ssh/192.168.1.2.pub ubuntu@192.168.1.2
ssh ubuntu@192.168.1.2

修改 sshd 設定

sudo vim /etc/ssh/sshd_config
PasswordAuthentication no
PubkeyAuthentication yes

PHP Dependency of phpMyadmin & WordPress

前置作業 – Nginx & PHP & MySQL

關於 MySQL 8 的安裝請參考這篇 – Ubuntu 安裝 MySQL 8.0 (mysql-community-server)

nginx 跟 php 是用 Ondřej Surý 打包好的。

sudo add-apt-repository -n ppa:ondrej/php && sudo add-apt-repository -y ppa:ondrej/nginx-mainline

MySQL 的部份看要使用 MySQLMariaDB 或是 PERCONA 都可以。

PHP7.4

sudo apt install dbconfig-common dbconfig-mysql javascript-common libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libcurl4 libflac8 libjs-cropper libjs-jquery libjs-sphinxdoc libjs-underscore liblua5.2-0 libnghttp2-14 libpcre2-8-0 libphp-phpmailer libpsl5 librtmp1 libsodium23 libzip4 php php-bz2 php-common php-curl php-gd php-getid3 php-mbstring php-mysql php-pear php-phpseclib php-tcpdf php-xml php-zip php7.4 php7.4-bz2 php7.4-cli php7.4-common php7.4-curl php7.4-fpm php7.4-gd php7.4-fpm php7.4-json php7.4-mbstring php7.4-mysql php7.4-opcache php7.4-readline php7.4-xml php7.4-zip publicsuffix vorbis-tools

PHP8.0(把上面 7.4 都換成 8.0,沒有的直接去掉 7.4)

sudo apt install php-getid3 php8.0 php8.0-bz2 php8.0-cli php8.0-common php8.0-curl php8.0-fpm php8.0-gd php8.0-fpm php8.0-mbstring php8.0-mysql php8.0-opcache php8.0-readline php8.0-xml php8.0-zip

phpMyAdmin 前製作業

sudo mysql_secure_installation

跑完之後,設定 phpmyadmin 帳號權限

sudo mysql -u root -p
GRANT ALL PRIVILEGES ON *.* TO 'phpmyadmin'@'localhost' WITH GRANT OPTION;
FLUSH PRIVILEGES;

設定 nginx

安裝 WordPress

Ubuntu Desktop 多網卡下使用 Network Manager 變更 route metric

nmcli 變更 metric

sudo nmcli connection modify $connection-name ipv4.route-metric 100
sudo nmcli connection down $connection-name
sudo nmcli connection up $connection-name

參考文章

How to set metric in OS moderated with NetworkManager

nmcli — command-line tool for controlling NetworkManager

2.3. 使用 NETWORKMANAGER 命令行工具 NMCLI

Ubuntu 安裝 MySQL 8.0 (mysql-community-server)

有鑑於 MySQL 8 的效能跟各方面的精進,還有安全上的保護,慢慢將手上的機器都換成 MySQL 8。

首先到 MySQL Community Downloads 挑選 MySQL APT Repository,然後下載 mysql-apt-config_0.8.16-1_all.deb,接下來安裝:

wget https://repo.mysql.com//mysql-apt-config_0.8.16-1_all.deb
dpkg -i mysql-apt-config_0.8.16-1_all.deb

然後更新 apt repository list

sudo apt update

安裝 mysql-community-client mysql-community-server

sudo apt install mysql-community-client mysql-community-server

察看安裝的版本

ubuntu@guestOS:~$ dpkg -l | grep mysql-community
ii  mysql-community-client           8.0.23-1ubuntu20.04                                                amd64        MySQL Client
ii  mysql-community-client-core      8.0.23-1ubuntu20.04                                                amd64        MySQL Client Core Binaries
ii  mysql-community-client-plugins   8.0.23-1ubuntu20.04                                                amd64        MySQL Client plugin
ii  mysql-community-server           8.0.23-1ubuntu20.04                                                amd64        MySQL Server
ii  mysql-community-server-core      8.0.23-1ubuntu20.04                                                amd64        MySQL Server Core Binaires

接下來編輯 /etc/mysqlmysql.conf.d/mysqld.cnf

sudo vim /etc/mysql/mysql.conf.d/mysqld.cnf

於 [mysqld] 之下,新增下列參數,不知道什麼原因,安裝好之後,只會預設 listen 在 IPv6

bind-address    = 127.0.0.1

重新啟動 MySQL

sudo systemctl restart mysql.service

接下來開始 MySQL 的設定。

sudo mysql_secure_installation

lxc/lxd container 建立與管理及佈署基本指令 – part. 0

查看有哪些 lxc

lxc list

查詢有哪些 image 可以安裝

lxc image list images: | grep -i ubuntu

建立 lxc 的 container

lxc launch images:ubuntu/focal/amd64 focal

啟用 lxc 的 container

lxc start focal

停用(關機) lxc 的 container

lxc stop focal

Container 的 info

lxc info focal

刪除 lxc 的 container

lxc delete focal

Login to your container

lxc exec $my-container bash

lxc 管理基本指令

# 查看有哪些 profile
lxc profile list
# 查看 profile - default 內容
lxc profile show default
# 編輯 profile - default 內容
lxc profile edit default

# 查看有哪些 network 可用
lxc network list
# 查看 br0 這個 network 內容
lxc network show br0

# 查看有哪些 storage
lxc storage list
# 查看 zfspool 這個 storage 內容
lxc storage show zfspool
# 編輯 zfspool 
lxc storage edit zfspool
# 建立 storage
lxc storage create pool1 zfs source=/dev/sdX zfs.pool_name=zfspool

Storage configuration

PHP 優化 – 啟用 HugePage (php7.4-fpm & php8.0-fpm)

啟用 Linux 系統的 HugePage(大型暫存分頁機制)可讓 PHP 的 Text 段落和記憶體中的分頁都改用 HugePage 來儲存,來減少 TLB(Translation Lookaside Buffer)遺失而提高效能。原本系統記憶體多以 4KB 來分頁,但啟用 HugePage 後則改以 2MB 來分頁。CPU 存取記憶體時,得查表才能得知虛擬定址和記憶體定址間的轉換,CPU 會透過 TLB 暫存來加速查表。若分頁單位越小,分頁筆數就越多,查表時就會得建立越多 TLB ,容易造成暫存資料的遺失而得重查,啟用了大型 HugePage 分頁設定就能減少 TLB 數量而降低遺失的問題。

若以小小站規模,透過sudo sysctl vm.nr_hugepages= 256 指令,來分配 256 個預留的大型暫存分頁(共 512MB)),然後在php.ini設定檔中加入 opcache.huge_code_pages=1的敘述就能啟用。

啟用 hugepages

sudo sysctl -w vm.nr_hugepages=256 && echo 'vm.nr_hugepages=256' | sudo tee -a /etc/sysctl.conf > /dev/null

編輯 php.ini 啟用功能

opcache.file_cache="/dev/shm"
opcache.huge_code_pages=1

觀察 hugepages 動態

ubuntu@host:~$ grep Huge /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:     256
HugePages_Free:      231
HugePages_Rsvd:       25
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:          524288 kB

php 使用的記憶體

ubuntu@host:~# size /usr/bin/php7.4
   text    data     bss     dec     hex filename
4061631  573374  119376 4754381  488bcd /usr/bin/php7.4

設定 Ubuntu 下 KVM 的 Bridge Networking

預設的 KVM 下,提供一個類 NAT 方式的 private network bridge,讓 VMs 可以跟外頭溝通,但是無法跟 host 溝通。但畢竟如果要方便的話當然是 guest VMs 可以跟 Host OS 直接透過 bridge 抽取(大量備份),速度跟方便性才會好。

我的環境是一台 Router,然後下面是 host OS(固定 ip),然後 Router 提供 DHCP 或固定 IP 給 guest VMs 使用,這樣網內互打速度才會快,被侷限的只有 Router 或是 Switch 的 backend 而已。以下有圖有真相,host 是 Ubuntu 18.04、guest VM 是 Ubuntu 20.04、網卡 virtio,選擇 e1000 或是 rtl8139 就不可能有這個速度了。

基本套件確認

sudo apt install qemu-system-x86 qemu-utils qemu-efi ovmf libvirt-clients libvirt-daemon-system virtinst bridge-utils

備註:qemu-efi 跟 ovmf 是如果 guest VMs 需要用 EFI 模式安裝(如 Windows 10)才使用。

關閉 bridge 的 netfilter

畢竟是要提昇內部網路的速度及降低 CPU 的使用量,所以把 bridge 的 netfilter 關閉。編輯或新增 /etc/sysctl.d/bridge.conf ,內容下:

net.bridge.bridge-nf-call-iptables=0
net.bridge.bridge-nf-call-ip6tables=0
net.bridge.bridge-nf-call-arptables=0

新增一個 udev 定義檔,告訴 kernel 說這個 bridge 不用 netfilter,定義檔:/etc/udev/rules.d/99-bridge.rules ,內容如下(整行沒斷句):

ACTION=="add", SUBSYSTEM=="module", KERNEL=="br_netfilter", RUN+="/sbin/sysctl -p /etc/sysctl.d/bridge.conf"

砍光原本 KVM 的網路界面

virsh net-destroy default
virsh net-undefine default

或是使用 ip 指令,以下預設安裝好的時候兩個界面分別是 virbr0 跟 virbr0-nic 。

ip link delete virbr0 type brigde
ip link delete virbr0-nic

建立新的 bridge 給 KVM 內的 guest VMs 用

我的環境如下:

編輯或新增 /etc/netplan/00-installer-config.yaml ,內容如下:

network:                                                                                                                        
    ethernets:
        enp4s0:
            dhcp4: false
            dhcp6: false
    bridges:
        br0:
            interfaces: [ enp4s0 ]
            addresses: [192.168.1.1/24]
            gateway4: 192.168.1.254
            mtu: 1500
            nameservers:
                addresses: [127.0.0.1]
            parameters:
                stp: true
                forward-delay: 4
            dhcp4: false
            dhcp6: true
    version: 2

接著使用 netplan 建立起這個 network bridge

sudo netplan apply

告訴 KVM 有這個 br0 可以當 Network Bridge 使用了

首先建立一個 XML 檔案,標注清楚要通知 KVM 哪些資訊,host-bridge.xml 檔案如下:

<network>
  <name>host-bridge</name>
  <forward mode="bridge"/>
  <bridge name="br0"/>
</network>

接著使用 virsh 通知 KVM 啟用這個 Network Bridge

sudo virsh net-define host-bridge.xml
sudo virsh net-start bridge
sudo virsh net-autostart bridge

查詢是否已啟用

j7@hostOS:~$ virsh net-list --all
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 host-bridge               active     yes           y

KVM 內 Guest OS 開啟後查看有哪些網路界面

j7@hostOS:~$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:0a brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 00:00:00:00:00:0a brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:00:00:00:00:0a brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.1/24 brd 192.168.1.255 scope global noprefixroute br0
       valid_lft forever preferred_lft forever
    inet6 2001:b011:****:****:****:****:****:d3da/64 scope global temporary dynamic 
       valid_lft 598sec preferred_lft 598sec
    inet6 fe80::2d8:61ff:fe2c:d70a/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
8: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq master br0 state UNKNOWN group default qlen 1000
    link/ether a00:00:00:0a:0a:0a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fe77:3785/64 scope link 

KVM 使用的 Guest Virtual Interfce

8: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq master br0 state UNKNOWN group default qlen 1000
link/ether a00:00:00:0a:0a:0a brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc54:ff:fe77:3785/64 scope link

收割 virtio-net 威能

ubuntu@guestOS:~$ iperf3 -c 192.168.1.1
Connecting to host 192.168.1.1, port 5201
[  5] local 192.168.1.30 port 60208 connected to 192.168.1.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.35 GBytes  46.0 Gbits/sec    0   3.14 MBytes       
[  5]   1.00-2.00   sec  5.38 GBytes  46.3 Gbits/sec    0   3.14 MBytes       
[  5]   2.00-3.00   sec  5.54 GBytes  47.5 Gbits/sec    0   3.14 MBytes       
[  5]   3.00-4.00   sec  5.36 GBytes  46.0 Gbits/sec    0   3.14 MBytes       
[  5]   4.00-5.00   sec  5.49 GBytes  47.1 Gbits/sec    0   3.14 MBytes       
[  5]   5.00-6.00   sec  5.62 GBytes  48.2 Gbits/sec    0   3.14 MBytes       
[  5]   6.00-7.00   sec  5.47 GBytes  47.0 Gbits/sec    0   3.14 MBytes       
[  5]   7.00-8.00   sec  5.29 GBytes  45.4 Gbits/sec    0   3.14 MBytes       
[  5]   8.00-9.00   sec  5.46 GBytes  46.9 Gbits/sec    0   3.14 MBytes       
[  5]   9.00-10.00  sec  5.33 GBytes  45.8 Gbits/sec    0   3.14 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  54.3 GBytes  46.6 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  54.3 GBytes  46.6 Gbits/sec                  receiver

參考文章:

Bridge Networking with KVM on Ubuntu

KVM: Creating a bridged network with NetPlan on Ubuntu bionic
How to Setup Bridge Networking with KVM on Ubuntu 20.04

Ubuntu 下使用 DNS over TLS (DoT)

DNS over TLS(縮寫:DoT)是通過傳輸層安全協定(TLS)來加密並打包域名系統(DNS)的安全協定。此協定旨在防止中間人攻擊與控制DNS資料以保護使用者隱私。引用自 Wikipedia – DNS over TLS,但是 DNS query 的效能會掉,而且是倍數的XD。

安裝及設定 stubby

以 Ubuntu 為例,設定檔位置為:/etc/stubby/stubby.yml (以下懶惰寫法):

resolution_type: GETDNS_RESOLUTION_STUB
dns_transport_list:
  - GETDNS_TRANSPORT_TLS
tls_authentication: GETDNS_AUTHENTICATION_REQUIRED
listen_addresses:
  - 192.168.2.11
  - 127.0.0.1
  -  0::1
round_robin_upstreams: 0
upstream_recursive_servers:
  - address_data: 1.1.1.1
    tls_auth_name: "cloudflare-dns.com"
  - address_data: 1.0.0.1
    tls_auth_name: "cloudflare-dns.com"
  - address_data: 2606:4700:4700::1111
    tls_auth_name: "cloudflare-dns.com"
  - address_data: 2606:4700:4700::1001
    tls_auth_name: "cloudflare-dns.com"

加密 key

跑一段時間之後 Stubby 會幫忙取得 sha256 的 key

修改 systemd-resolve

sudo vim /etc/systemd/resolved.conf
--------
#  This file is part of systemd.                                                                                                
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See resolved.conf(5) for details

[Resolve]
DNS=127.0.0.1
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#DNSSEC=yes
#Cache=yes
#DNSStubListener=yes

驗證(收割)DoT

dig @127.0.0.1 www.google.com

; <<>> DiG 9.11.3-1ubuntu1.13-Ubuntu <<>> @127.0.0.1 www.google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16028
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; PAD: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 (".....................................................................................................................................................................................................................................................................................................................................................................................................................")
;; QUESTION SECTION:
;www.google.com.			IN	A

;; ANSWER SECTION:
www.google.com.		60	IN	A	172.217.160.100

;; Query time: 5 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Tue Dec 22 23:18:47 CST 2020
;; MSG SIZE  rcvd: 482

DoT 連 853 port 狀況

netstat -tuWc | grep 853
tcp        0      0 host:88387         one.one.one.one:853     ESTABLISHED
tcp        0      0 host:38788         one.one.one.one:853     ESTABLISHED
tcp6       0      0 2001-b011-0000-0000-0000-0-0-0.dynamic-ip6.hinet.net:54386 one.one.one.one:853     ESTABLISHED
tcp6       0      0 2001-b011-0000-0000-0000-0-0-0.dynamic-ip6.hinet.net:38372 one.one.one.one:853     ESTABLISHED