Server Provisioning
This is intended to be a reference for setting up a VM or Server for AllStatLink.
Server Overview
Basic Requirements
AllStarLink has standardized on Ubuntu 16.04 LTS for it's servers.
The minimum configuration of any server will be 2 cores of 2 GHz or faster, 4 GiB of ram and 40 GiB of Disk.
All new servers shall support IPv6, or have it available from the hosting provider.
Reverse IP's shall be delegated via a CNAME to $NAME.PTR.allstarlink.org, where $NAME is the name of the server.
All servers shall be partitioned to use / as the only partition unless a specific configuration is required.
Install guide
When provisioning a new server
- check the VM is setup (cpu/mem/disk) as it should be, if not contact the provider
cat /proc/cpuinfo |grep processor processor : 0 processor : 1 processor : 2 processor : 3
- on the server install python (apt-get install python). This is needed for the ansibile provisioning
- setup the server in the infrastructure configs and push the users and keys to it.
Mandatory Software
All servers require this software
apt-get install ntp python vim screen ipsec-tools strongswan fail2ban snmp
Mandatory Configs
- The network should be configured to use /etc/network/interfaces, and add DNS and the firewall to it and search in the allstarlink.org domain
# The primary network interface
auto eth0
iface eth0 inet static
address 44.103.0.49
netmask 255.255.255.0
network 44.103.0.0
broadcast 44.103.0.255
gateway 44.103.0.1
dns-nameservers 44.103.0.4 1.1.1.1
dns-search allstarlink.org
up /etc/network/firewall.sh
- There is typically only one network interface, and it will be named dynamically. We must setup this using udev to be persistent
root@server# ifconfig |grep HWaddr eth0 Link encap:Ethernet HWaddr 52:54:00:73:86:06
Now take this HWaddr and put it in the config file
echo 'SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:73:86:06", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="eth0"' >/etc/udev/rules.d/70-persistent-net.rules
- configure screen to use the scroll back buffer
vim /etc/screenrc uncomment "termcapinfo xterm|xterms|xs|rxvt ti@:te@"
- configure bash completion for interactive shells
vim /etc/bash.bashrc uncomment the stuff below # enable bash completion in interactive shells
- set the host name
echo "domain.allstarlink.org" >/etc/hostname
- set the default editor
update-alternatives --config editor Then select #3 vim.basic
- setup a firewall as /etc/network/firewall.sh and chmod +x it. You'll need to edit this based on the machine. Note the stuff in tampa uses a firewall on the HV too.
#!/bin/bash
INET_IF=eth0
#Flush and zero all tables
modprobe ip_tables
modprobe ipt_limit
modprobe iptable_mangle
modprobe ipt_state
modprobe ipt_LOG
modprobe iptable_filter
iptables -F INPUT
iptables -F FORWARD
iptables -t nat -F POSTROUTING
iptables -t nat -F PREROUTING
#init the log-and-drop chain
iptables -F log-and-drop
iptables -X log-and-drop
iptables -N log-and-drop
#init log-and-reject
iptables -F log-and-reject
iptables -X log-and-reject
iptables -N log-and-reject
echo "all tables flushed and dropped"
# Specific chain used for logging packets before blocking them
iptables -A log-and-drop -j LOG --log-prefix "[IPTables] Drop "
iptables -A log-and-drop -j DROP
# Specific chain used for logging packets before blocking them
iptables -A log-and-reject -j LOG --log-prefix "[IPTables] Reject "
iptables -A log-and-reject -j REJECT
echo "logging chains setup"
# The packets having the TCP flags activated are dropped
# and so for the ones with no flag at all (often used with Nmap scans)
iptables -A FORWARD -p tcp --tcp-flags ALL ALL -j log-and-drop
iptables -A FORWARD -p tcp --tcp-flags ALL NONE -j log-and-drop
#Global blocks
#iptables -t filter -A INPUT -j DROP -s 119.118.232.185/24
# allow IPSEC from other boxes
IPSECsrc='199.47.174.150,44.98.254.151,44.103.0.48,44.103.0.49,44.98.254.145,44.72.21.13,44.72.21.12'
#Technically the next two are not needed as we have the policy
iptables -A INPUT -i $INET_IF -p 50 -j ACCEPT --src "$IPSECsrc"
iptables -A INPUT -i $INET_IF -p 51 -j ACCEPT --src "$IPSECsrc"
iptables -A INPUT -i $INET_IF -p udp --dport 500 -j ACCEPT --src "$IPSECsrc"
# this is needed to allow all ipsec packets when it's host to host
iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc"
# allow all ssh in
iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 22
#allow http and https
#iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 80
#iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 443
# allow asterisk 4569
#iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 4569
# allow DNS
#iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 53
#iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 53
echo "end of services"
# allow ping at 2 per sec
iptables -t filter -A INPUT -j ACCEPT --in-interface $INET_IF --protocol icmp --icmp-type echo-request --match limit --limit 4/s --limit-burst 3
iptables -t filter -A INPUT -j log-and-drop --in-interface $INET_IF --protocol icmp --icmp-type echo-request
# allow responces to local initated connections
#iptables -A INPUT -i $INET_IF --match state --state NEW,INVALID -j log-and-drop
#iptables -A FORWARD -i $INET_IF --match state --state NEW,INVALID -j log-and-drop
iptables -t filter -A INPUT -j ACCEPT --match state --state RELATED,ESTABLISHED
# Set rp_filter to 2
for i in `find /proc/sys/net/ipv*/conf -name rp_filter`
do
echo "2" >$i
done
# setup a default deny rule for outside traffic
iptables -t filter -A INPUT --in-interface $INET_IF -j log-and-drop
- setup fail2ban for ssh and have it null route offenders. edit ignoreip as needed
vi /etc/fail2ban/jail.conf ignoreip = 127.0.0.1/8 199.47.172.0/22 44.98.254.0/24 44.72.21.0/24 44.103.0.0/24 bantime = 3600 # A host is banned if it has generated "maxretry" during the last "findtime" # seconds. findtime = 3600 # "maxretry" is the number of failures before a host get banned. maxretry = 2 banaction = route
- Set the TimeZone to UTC
sudo timedatectl set-timezone UTC
- Set the server up in forward and reverse DNS
- for reverse have the provider do a CNAME in their reverse file pointing to $DOMAIN.PTR.allstarlink.org. In the allstarlink.org DNS zone add an entry
example: stats IN PTR stats.allstarlink.org.
This will do a lookup on 130.254.98.44.in-addr.arpa. and return a CNAME pointing to stats.PTR.allstarlink.org, which has a PTR record pointing to stats.allstarlink.org.
Configure IPSEC
AllStarLink servers use strong crypto using host to host IPSEC between them for protection of services. This is configured only between servers that need it, as we don't have dynamic tunneling enabled, and each server needs a config for each tunnel. This can quickly add up to lots of configrations.
This example will show two servers, 1 and 2 with IP 44.1.1.1 and 44.2.2.2 respectively.
Server 1
We need to provision the ipsec tools to know about the connections and configure a pre shared key (PSK). Note the left server is always the local server.
/etc/ipsec.conf
conn one-to-two
authby=secret
#auto=start enabled the tunnel to come up even if there is not traffic for it.
auto=start
keyexchange=ike
left=4.1.1.1
right=4.2.2.2
leftikeport=500
rightikeport=500
type=transport
esp=aes128gcm16!
dpddelay=5
dpdtimeout=20
dpdaction=restart
vim /etc/ipsec.secrets 44.1.1.1 44.2.2.2 : PSK "This is the AllStarLink PSK"
Then do an 'ipsec restart' on the server.
Server 2
/etc/ipsec.conf
conn two-to-one
authby=secret
#auto=start enabled the tunnel to come up even if there is not traffic for it.
auto=start
keyexchange=ike
left=4.2.2.2
right=4.1.1.1
leftikeport=500
rightikeport=500
type=transport
esp=aes128gcm16!
dpddelay=5
dpdtimeout=20
dpdaction=restart
vim /etc/ipsec.secrets 44.2.2.2 44.1.1.1 : PSK "This is the AllStarLink PSK"
Then do an 'ipsec restart' on the server.
Verify IPsec
The 'ipsec' command is used to verify the tunnel is up between the servers
root@server# ipsec status
two-to-one[839]: ESTABLISHED 98 minutes ago, 44.1.1.1[44.1.1.1]...44.2.2.2[44.2.2.2]
two-to-one{13209}: INSTALLED, TRANSPORT, reqid 695, ESP SPIs: c824e4db_i c1e4bf5c_o
two-to-one{13209}: 44.1.1.1/32 === 44.2.2.2/32
If they are not up, check /var/log/syslog and restart ipsec on both servers. Some times a server can get in a bad status if there is a mis-config. Also it's worth noting that IPSEC is processed by iptables once it's decrypted, the iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc" line in the firewall allows all IPsec packets once decrypted to bypass the firewall. This is able to prevent traffic between unencrypted services on the servers (e.g. mysql will not connect if the ipsec is down).
Verification
It's important to verify the server provisiong before being put into production.
Items to check
- reboot the server/vm, do all services start properly?
- Is the IP address configured on the server on eth0?
- Is the hostname set?
- Is it configured in DNS both forward and reverse?
- Is the firewall active (try netcat on a non-permitted port)
- IPSEC is active
ipsec status? - Does Screen work in an xterm with scroll back?
- Is the time set via ntp
ntpdateand is the timezone set to UTC? - Is fail2ban working? Make a couple test connections and see if the IP is null routed
ip route show
You may need to check your other services on this server now.
Network Monitoring
It's time to hand off the server to the NMS team. Please ensure SNMP is configured and an IPSEC tunnel is built to nms.allstarlink.org Logging will be sucked up by graylog.
Please ensure it's being watched in librenms by asking on the admin list or in the slack.