Server Provisioning
This is intended to be a reference for setting up a VM or Server for AllStatLink.
Server Overview
Basic Requirements
AllStarLink has standardized on Ubuntu 16.04 LTS for it's servers.
The minimum configuration of any server will be 2 cores of 2 GHz or faster, 4 GiB of ram and 40 GiB of Disk.
All new servers shall support IPv6, or have it available from the hosting provider.
Reverse IP's shall be delegated via a CNAME to $NAME.PTR.allstarlink.org, where $NAME is the name of the server.
All servers shall be partitioned to use / as the only partition unless a specific configuration is required.
Install guide
When provisioning a new server
- check the VM is setup (cpu/mem/disk) as it should be, if not contact the provider
cat /proc/cpuinfo |grep processor processor : 0 processor : 1 processor : 2 processor : 3
- on the server install python (apt-get install python). This is needed for the ansibile provisioning
- setup the server in the infrastructure configs and push the users and keys to it.
Mandatory Software
All servers require this software
apt-get install ntp ntpdate python vim screen ipsec-tools strongswan fail2ban snmp haveged libacl1-dev python3-dev libssl-dev gcc g++ fio pbzip2 ncdu
Mandatory Configs
Ubuntu 18 Config
Ubuntu 18 uses the net netplan config. It's gay as fuck.
First you need to disable the resolved service:
sudo systemctl disable systemd-resolved.service sudo systemctl stop systemd-resolved rm /etc/resolv.conf sudo touch /etc/cloud/cloud-init.disabled sudo apt-get purge cloud-init
echo "nameserver 1.1.1.1" > /etc/resolv.conf apt-get install ifupdown
Configure /etc/network/interfaces
systemctl unmask networking systemctl enable networking systemctl restart networking
systemctl stop systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online systemctl disable systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online systemctl mask systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online apt-get --assume-yes purge nplan netplan.io
Network Config
- The network should be configured to use /etc/network/interfaces, and add DNS and the firewall to it and search in the allstarlink.org domain
# The primary network interface auto eth0 iface eth0 inet6 static address 9805:0900:0340:1000::2600/64 autoconf 0 accept_ra 2 iface eth0 inet static address 44.103.0.49 netmask 255.255.255.0 network 44.103.0.0 broadcast 44.103.0.255 gateway 44.103.0.1 dns-nameservers 44.103.0.4 1.1.1.1 dns-search allstarlink.org up /etc/network/firewall.sh
Persistent Interface Names
UDEV method
- There is typically only one network interface, and it will be named dynamically. We must setup this using udev to be persistent
root@server# ifconfig |grep HWaddr eth0 Link encap:Ethernet HWaddr 52:54:00:73:86:06
Now take this HWaddr and put it in the config file
echo 'SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:73:86:06", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="eth0"' >/etc/udev/rules.d/70-persistent-net.rules
Grub method
- Edit the /etc/default/grub file:
vim /etc/default/grub
- Look for "GRUB_CMDLINE_LINUX" and add the following "net.ifnames=0 biosdevname=0"
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"
- Generate new grub config file
sudo grub-mkconfig -o /boot/grub/grub.cfg
Other required config items
- configure screen to use the scroll back buffer
vim /etc/screenrc uncomment "termcapinfo xterm|xterms|xs|rxvt ti@:te@"
- configure bash completion for interactive shells
vim /etc/bash.bashrc uncomment the stuff below # enable bash completion in interactive shells
- set the host name
echo "domain.allstarlink.org" >/etc/hostname
- set the default editor
update-alternatives --config editor Then select #3 vim.basic
- setup a firewall as /etc/network/firewall.sh and chmod +x it. You'll need to edit this based on the machine. Note the stuff in tampa uses a firewall on the HV too.
#!/bin/bash INET_IF=eth0 #Flush and zero all tables modprobe ip_tables modprobe ipt_limit modprobe iptable_mangle modprobe ipt_state modprobe ipt_LOG modprobe iptable_filter iptables -F INPUT iptables -F FORWARD iptables -t nat -F POSTROUTING iptables -t nat -F PREROUTING #init the log-and-drop chain iptables -F log-and-drop iptables -X log-and-drop iptables -N log-and-drop #init log-and-reject iptables -F log-and-reject iptables -X log-and-reject iptables -N log-and-reject echo "all tables flushed and dropped" # Specific chain used for logging packets before blocking them iptables -A log-and-drop -j LOG --log-prefix "[IPTables] Drop " iptables -A log-and-drop -j DROP # Specific chain used for logging packets before blocking them iptables -A log-and-reject -j LOG --log-prefix "[IPTables] Reject " iptables -A log-and-reject -j REJECT echo "logging chains setup" # The packets having the TCP flags activated are dropped # and so for the ones with no flag at all (often used with Nmap scans) iptables -A FORWARD -p tcp --tcp-flags ALL ALL -j log-and-drop iptables -A FORWARD -p tcp --tcp-flags ALL NONE -j log-and-drop #Global blocks #iptables -t filter -A INPUT -j DROP -s 119.118.232.185/24 # allow IPSEC from other boxes IPSECsrc='199.47.174.150,44.98.254.151,44.103.0.48,44.103.0.49,44.98.254.145,44.72.21.13,44.72.21.12' #Technically the next two are not needed as we have the policy iptables -A INPUT -i $INET_IF -p 50 -j ACCEPT --src "$IPSECsrc" iptables -A INPUT -i $INET_IF -p 51 -j ACCEPT --src "$IPSECsrc" iptables -A INPUT -i $INET_IF -p udp --dport 500 -j ACCEPT --src "$IPSECsrc" # this is needed to allow all ipsec packets when it's host to host iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc" # allow all ssh in iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 22 #allow http and https #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 80 #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 443 # allow asterisk 4569 #iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 4569 # allow DNS #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 53 #iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 53 echo "end of services" # allow ping at 2 per sec iptables -t filter -A INPUT -j ACCEPT --in-interface $INET_IF --protocol icmp --icmp-type echo-request --match limit --limit 4/s --limit-burst 3 iptables -t filter -A INPUT -j log-and-drop --in-interface $INET_IF --protocol icmp --icmp-type echo-request # allow responces to local initated connections #iptables -A INPUT -i $INET_IF --match state --state NEW,INVALID -j log-and-drop #iptables -A FORWARD -i $INET_IF --match state --state NEW,INVALID -j log-and-drop iptables -t filter -A INPUT -j ACCEPT --match state --state RELATED,ESTABLISHED # Set rp_filter to 2 for i in `find /proc/sys/net/ipv*/conf -name rp_filter` do echo "2" >$i done # setup a default deny rule for outside traffic iptables -t filter -A INPUT --in-interface $INET_IF -j log-and-drop
- setup fail2ban for ssh and have it null route offenders. edit ignoreip as needed
vi /etc/fail2ban/jail.conf ignoreip = 127.0.0.1/8 199.47.172.0/22 44.98.254.0/24 44.72.21.0/24 44.103.0.0/24 bantime = 3600 # A host is banned if it has generated "maxretry" during the last "findtime" # seconds. findtime = 3600 # "maxretry" is the number of failures before a host get banned. maxretry = 2 banaction = route
- Set the TimeZone to UTC
sudo timedatectl set-timezone UTC
- Set the server up in forward and reverse DNS
- for reverse have the provider do a CNAME in their reverse file pointing to $DOMAIN.PTR.allstarlink.org. In the allstarlink.org DNS zone add an entry
example: stats IN PTR stats.allstarlink.org.
This will do a lookup on 130.254.98.44.in-addr.arpa. and return a CNAME pointing to stats.PTR.allstarlink.org, which has a PTR record pointing to stats.allstarlink.org.
Configure IPSEC
AllStarLink servers use strong crypto using host to host IPSEC between them for protection of services. This is configured only between servers that need it, as we don't have dynamic tunneling enabled, and each server needs a config for each tunnel. This can quickly add up to lots of configrations.
This example will show two servers, 1 and 2 with IP 44.1.1.1 and 44.2.2.2 respectively.
Server 1
We need to provision the ipsec tools to know about the connections and configure a pre shared key (PSK). Note the left server is always the local server.
/etc/ipsec.conf conn one-to-two authby=secret #auto=start enabled the tunnel to come up even if there is not traffic for it. auto=start keyexchange=ike left=4.1.1.1 right=4.2.2.2 leftikeport=500 rightikeport=500 type=transport esp=aes128gcm16! dpddelay=5 dpdtimeout=20 dpdaction=restart
vim /etc/ipsec.secrets 44.1.1.1 44.2.2.2 : PSK "This is the AllStarLink PSK"
Then do an 'ipsec restart' on the server.
Server 2
/etc/ipsec.conf
conn two-to-one authby=secret #auto=start enabled the tunnel to come up even if there is not traffic for it. auto=start keyexchange=ike left=4.2.2.2 right=4.1.1.1 leftikeport=500 rightikeport=500 type=transport esp=aes128gcm16! dpddelay=5 dpdtimeout=20 dpdaction=restart
vim /etc/ipsec.secrets 44.2.2.2 44.1.1.1 : PSK "This is the AllStarLink PSK"
Then do an 'ipsec restart' on the server.
Verify IPsec
The 'ipsec' command is used to verify the tunnel is up between the servers
root@server# ipsec status two-to-one[839]: ESTABLISHED 98 minutes ago, 44.1.1.1[44.1.1.1]...44.2.2.2[44.2.2.2] two-to-one{13209}: INSTALLED, TRANSPORT, reqid 695, ESP SPIs: c824e4db_i c1e4bf5c_o two-to-one{13209}: 44.1.1.1/32 === 44.2.2.2/32
If they are not up, check /var/log/syslog and restart ipsec on both servers. Some times a server can get in a bad status if there is a mis-config. Also it's worth noting that IPSEC is processed by iptables once it's decrypted, the iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc"
line in the firewall allows all IPsec packets once decrypted to bypass the firewall. This is able to prevent traffic between unencrypted services on the servers (e.g. mysql will not connect if the ipsec is down).
Configure Postfix
Postfix is installed to forward mail for root to a smtp host.
apt-get install postfix mailutils
This will run an installer with a curses interface and you must select Satallite System. Check the System mail name is the hostname of the server, and the SMTP relay host is morty.keekles.org. Root and postmaster mail should be rootmail@allstarlink.org.
Should you need to reconfigure this use:
dpkg-reconfigure postfix
other aliases are setup in /etc/aliases. You must run newaliases after this is updated for them to take effect.
Verification
It's important to verify the server provisiong before being put into production.
Items to check
- reboot the server/vm, do all services start properly?
- Is the IP address configured on the server on eth0?
- Is the hostname set?
- Is it configured in DNS both forward and reverse?
- Is the firewall active (try netcat on a non-permitted port)
- IPSEC is active
ipsec status
? - Does Screen work in an xterm with scroll back?
- Is the time set via ntp
ntptime
and is the timezone set to UTC? - Is fail2ban working? Make a couple test connections and see if the IP is null routed
ip route show
You may need to check your other services on this server now.
Network Monitoring
It's time to hand off the server to the NMS team. Please ensure SNMP is configured and an IPSEC tunnel is built to nms.allstarlink.org Logging will be sucked up by graylog.
Please ensure it's being watched in librenms by asking on the admin list or in the slack.