Changes

Jump to navigation Jump to search

Server Provisioning (view source)

Revision as of 05:48, 24 January 2022

12,272 bytes added ,  2 years ago
m
no edit summary
Line 1: Line 1: −
This is intended to be a reference for setting up a VM or Server for AllStatLink.
+
[[Category: Infrastructure]]
 +
[[Category: How to]]
 +
{{Notice | This document reflects the current Infrastructure as of 2021-01-17}}
 +
This is intended to be a reference for setting up a VM or Server for PTTLink.  
    
= Server Overview =  
 
= Server Overview =  
Line 34: Line 37:  
All servers require this software
 
All servers require this software
   −
apt-get install ntp python vim screen ipsec-tools strongswan fail2ban  
+
apt-get install ntp ntpdate python vim screen ipsec-tools strongswan fail2ban snmp haveged libacl1-dev python3-dev libssl-dev gcc g++ fio pbzip2 ncdu
   −
=== Mandatory Configs ===
+
=== Configuration ===
 +
There are two types of configuration presented below:
 +
*No Netplan config - removes netplan and swtiches back to ifupdown
 +
*Netplan config - keeps netplan
    +
As new versions of Ubuntu are released, it is very possible that netplan will become the only officially supported means to configure networking.  Keeping that in mind, all efforts should be made to configure Ubuntu 20+ servers using netlpan with a fallback to the no netplan config as the last resort.
 +
 +
==== No Netplan Config ====
 +
 +
You can remove this and go back to ''ifupdown'' as follows.
 +
 +
First you need to disable the resolved service:
 +
 +
sudo systemctl disable systemd-resolved.service
 +
sudo systemctl stop systemd-resolved
 +
rm /etc/resolv.conf
 +
sudo touch /etc/cloud/cloud-init.disabled
 +
sudo apt-get purge cloud-init
 +
 +
echo "nameserver 1.1.1.1" > /etc/resolv.conf
 +
apt-get install ifupdown
 +
 +
Reconfigure  network services
 +
 +
systemctl unmask networking
 +
systemctl enable networking
 +
systemctl restart networking
 +
 +
systemctl stop systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online
 +
systemctl disable systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online
 +
systemctl mask systemd-networkd.socket systemd-networkd networkd-dispatcher systemd-networkd-wait-online
 +
apt-get --assume-yes purge nplan netplan.io
 +
 +
Network Config
 
* The network should be configured to use /etc/network/interfaces, and add DNS and the firewall to it and search in the allstarlink.org domain
 
* The network should be configured to use /etc/network/interfaces, and add DNS and the firewall to it and search in the allstarlink.org domain
   −
    # The primary network interface
+
# The primary network interface
−
    auto eth0
+
auto eth0
−
     iface eth0 inet static
+
iface eth0 inet6 static
−
            address 44.103.0.49
+
     address 9805:0900:0340:1000::2600/64
−
            netmask 255.255.255.0
+
    autoconf 0
−
            network 44.103.0.0
+
    accept_ra 2
−
            broadcast 44.103.0.255
+
iface eth0 inet static
−
            gateway 44.103.0.1  
+
    address 44.103.0.49
−
            dns-nameservers 44.103.0.4 1.1.1.1
+
    netmask 255.255.255.0
−
            dns-search allstarlink.org
+
    network 44.103.0.0
−
            up /etc/network/firewall.sh
+
    broadcast 44.103.0.255
 +
    gateway 44.103.0.1  
 +
    dns-nameservers 44.103.0.4 1.1.1.1
 +
    dns-search allstarlink.org
 +
    up /etc/network/firewall.sh
 +
 
 +
==== Netplan config ====
 +
 
 +
Configure the network using /etc/netplan files.  You should remove any existing files and create a new one called 01-netcfg.yaml with the following:
    +
  network:
 +
    version: 2
 +
    rendered: networkd
 +
    ethernets:
 +
      eth0:
 +
        addresses:
 +
          - 44.98.254.1/24
 +
        gateway4: 44.98.254.1
 +
        nameservers:
 +
          search: [allstarlink.org]
 +
          addresses: [1.1.1.1]
 +
       
 +
*Once done run the following and test to make sure the IP address is reachable.  Follow the on-screen instructions:
 +
  netplan try
    +
Refer to [https://netplan.io/examples/ Netplan configuration examples] for more examples on how to configure networking using netplan
   −
* There is typically only one network interface, and it will be named dynamically.  We must setup this using udev to be persistant
+
==== Persistent Interface Names ====
 +
 
 +
To ensure that interface names are persistent (e.g., ethX) you can choose using either UDEV or Grub. 
 +
 
 +
The Grub method tends to be easier and less prone to locking yourself out of a VM due to an interface naming conflict at reboot.
 +
 
 +
====== UDEV method ======
 +
* There is typically only one network interface, and it will be named dynamically.  We must setup this using udev to be persistent
    
     root@server# ifconfig |grep HWaddr
 
     root@server# ifconfig |grep HWaddr
Line 61: Line 126:  
Now take this HWaddr and put it in the config file
 
Now take this HWaddr and put it in the config file
 
     echo 'SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:73:86:06", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="eth0"' >/etc/udev/rules.d/70-persistent-net.rules
 
     echo 'SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="52:54:00:73:86:06", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME="eth0"' >/etc/udev/rules.d/70-persistent-net.rules
 +
 +
====== Grub method ======
 +
* Edit the '''''/etc/default/grub''''' file:
 +
  vim /etc/default/grub
 +
 +
* Look for "GRUB_CMDLINE_LINUX" and add the following "net.ifnames=0 biosdevname=0"
 +
  GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"
 +
 +
* Generate new grub config file
 +
  sudo grub-mkconfig -o /boot/grub/grub.cfg
 +
 +
==== Other config items ====
 +
 +
* configure screen to use the scroll back buffer
 +
  vim /etc/screenrc
 +
  uncomment "termcapinfo xterm|xterms|xs|rxvt ti@:te@"
 +
 +
* configure bash completion for interactive shells
 +
    vim /etc/bash.bashrc
 +
    uncomment the stuff below
 +
    # enable bash completion in interactive shells
 +
 +
* set the host name
 +
    echo "domain.allstarlink.org" >/etc/hostname
 +
 +
* set the default editor
 +
    update-alternatives --config editor
 +
    Then select #3 vim.basic
 +
 +
* setup a firewall and chmod +x it.  You'll need to edit this based on the machine.
 +
**For netplan place this file in '''''/etc/networkd-dispatcher/routable.d/50-ifup-hooks'''''
 +
**For ifupdown place this file in '''''/etc/network/firewall.sh'''''
 +
 +
    #!/bin/bash
 +
   
 +
    INET_IF=eth0
 +
   
 +
    #Flush and zero all tables
 +
    modprobe ip_tables
 +
    modprobe ipt_limit
 +
    modprobe iptable_mangle
 +
    modprobe ipt_state
 +
    modprobe ipt_LOG
 +
    modprobe iptable_filter
 +
   
 +
    iptables -F INPUT
 +
    iptables -F FORWARD
 +
    iptables -t nat -F POSTROUTING
 +
    iptables -t nat -F PREROUTING
 +
   
 +
    #init the log-and-drop chain
 +
    iptables -F log-and-drop
 +
    iptables -X log-and-drop
 +
    iptables -N log-and-drop
 +
    #init log-and-reject
 +
    iptables -F log-and-reject
 +
    iptables -X log-and-reject
 +
    iptables -N log-and-reject
 +
   
 +
    echo "all tables flushed and dropped"
 +
    # Specific chain used for logging packets before blocking them
 +
    iptables -A log-and-drop -j LOG --log-prefix "[IPTables] Drop "
 +
    iptables -A log-and-drop -j DROP
 +
   
 +
    # Specific chain used for logging packets before blocking them
 +
    iptables -A log-and-reject -j LOG --log-prefix "[IPTables] Reject "
 +
    iptables -A log-and-reject -j REJECT
 +
    echo "logging chains setup"
 +
   
 +
    # The packets having the TCP flags activated are dropped
 +
    # and so for the ones with no flag at all (often used with Nmap scans)
 +
    iptables -A FORWARD -p tcp --tcp-flags ALL ALL -j log-and-drop
 +
    iptables -A FORWARD -p tcp --tcp-flags ALL NONE -j log-and-drop
 +
   
 +
    #Global blocks
 +
    #iptables -t filter -A INPUT -j DROP -s 119.118.232.185/24
 +
   
 +
    # allow IPSEC from other boxes
 +
    IPSECsrc='199.47.174.150,44.98.254.151,44.103.0.48,44.103.0.49,44.98.254.145,44.72.21.13,44.72.21.12'
 +
    #Technically the next two are not needed as we have the policy
 +
    iptables -A INPUT -i $INET_IF -p 50 -j ACCEPT --src "$IPSECsrc"
 +
    iptables -A INPUT -i $INET_IF -p 51 -j ACCEPT --src "$IPSECsrc"
 +
    iptables -A INPUT -i $INET_IF -p udp --dport 500 -j ACCEPT --src "$IPSECsrc"
 +
    # this is needed to allow all ipsec packets when it's host to host
 +
    iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc"
 +
   
 +
    # allow all ssh in
 +
    iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 22
 +
   
 +
    #allow http and https
 +
    #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 80
 +
    #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 443
 +
    # allow asterisk 4569
 +
    #iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 4569
 +
    # allow DNS
 +
    #iptables -t filter -A INPUT -j ACCEPT --protocol tcp --dport 53
 +
    #iptables -t filter -A INPUT -j ACCEPT --protocol udp --dport 53
 +
   
 +
    echo "end of services"
 +
    # allow ping at 2 per sec
 +
    iptables -t filter -A INPUT -j ACCEPT --in-interface $INET_IF --protocol icmp --icmp-type echo-request --match limit --limit 4/s --limit-burst 3
 +
    iptables -t filter -A INPUT -j log-and-drop  --in-interface $INET_IF --protocol icmp --icmp-type echo-request
 +
    # allow responces to local initated connections
 +
    #iptables -A INPUT -i  $INET_IF --match state --state NEW,INVALID -j log-and-drop
 +
    #iptables -A FORWARD -i $INET_IF  --match state --state NEW,INVALID -j log-and-drop
 +
    iptables -t filter -A INPUT -j ACCEPT --match state --state RELATED,ESTABLISHED
 +
    # Set rp_filter to 2
 +
    for i in `find /proc/sys/net/ipv*/conf -name rp_filter`
 +
    do
 +
            echo "2" >$i
 +
    done
 +
    # setup a default deny rule for outside traffic
 +
    iptables -t filter -A INPUT --in-interface $INET_IF -j log-and-drop
 +
 +
* setup fail2ban for ssh and have it null route offenders. edit ignoreip as needed
 +
    vi /etc/fail2ban/jail.conf
 +
    ignoreip = 127.0.0.1/8 199.47.172.0/22 44.98.254.0/24 44.72.21.0/24 44.103.0.0/24
 +
    bantime  = 3600
 +
    # A host is banned if it has generated "maxretry" during the last "findtime"
 +
    # seconds.
 +
    findtime  = 3600
 +
    # "maxretry" is the number of failures before a host get banned.
 +
    maxretry = 2
 +
    banaction = route
 +
 +
* Set the TimeZone to UTC
 +
    sudo timedatectl set-timezone UTC
 +
* Set the server up in forward and reverse DNS
 +
** for reverse have the provider do a CNAME in their reverse file pointing to $DOMAIN.PTR.allstarlink.org.  In the allstarlink.org DNS zone add an entry
 +
    example:
 +
    stats            IN      PTR    stats.allstarlink.org.
 +
This will do a lookup on 130.254.98.44.in-addr.arpa. and return a CNAME pointing to stats.PTR.allstarlink.org, which has a PTR record pointing to stats.allstarlink.org.
 +
 +
=== Configure IPSEC ===
 +
AllStarLink servers use strong crypto using host to host IPSEC between them for protection of services.  This is configured only between servers that need it, as we don't have dynamic tunneling enabled, and each server needs a config for each tunnel.  This can quickly add up to lots of configrations.
 +
 +
This example will show two servers, 1 and 2 with IP 44.1.1.1 and 44.2.2.2 respectively.
 +
 +
==== Server 1 ====
 +
We need to provision the ipsec tools to know about the connections and configure a pre shared key (PSK).
 +
Note the left server is always the local server.
 +
 +
    /etc/ipsec.conf
 +
    conn one-to-two
 +
            authby=secret
 +
            #auto=start enabled the tunnel to come up even if there is not traffic for it. 
 +
            auto=start
 +
            keyexchange=ike
 +
            left=4.1.1.1
 +
            right=4.2.2.2
 +
            leftikeport=500
 +
            rightikeport=500 
 +
            type=transport
 +
            esp=aes128gcm16!
 +
            dpddelay=5
 +
            dpdtimeout=20
 +
            dpdaction=restart
 +
 +
    vim /etc/ipsec.secrets
 +
    44.1.1.1 44.2.2.2 : PSK "This is the AllStarLink PSK"
 +
Then do an 'ipsec restart' on the server.
 +
 +
==== Server 2 ====
 +
 +
/etc/ipsec.conf
 +
    conn two-to-one
 +
            authby=secret
 +
            #auto=start enabled the tunnel to come up even if there is not traffic for it. 
 +
            auto=start
 +
            keyexchange=ike
 +
            left=4.2.2.2
 +
            right=4.1.1.1
 +
            leftikeport=500
 +
            rightikeport=500 
 +
            type=transport
 +
            esp=aes128gcm16!
 +
            dpddelay=5
 +
            dpdtimeout=20
 +
            dpdaction=restart
 +
 +
    vim /etc/ipsec.secrets
 +
    44.2.2.2 44.1.1.1 : PSK "This is the AllStarLink PSK"
 +
Then do an 'ipsec restart' on the server.
 +
 +
==== Verify IPsec ====
 +
 +
The 'ipsec' command is used to verify the tunnel is up between the servers
 +
 +
    root@server# ipsec status
 +
    two-to-one[839]: ESTABLISHED 98 minutes ago, 44.1.1.1[44.1.1.1]...44.2.2.2[44.2.2.2]
 +
    two-to-one{13209}:  INSTALLED, TRANSPORT, reqid 695, ESP SPIs: c824e4db_i c1e4bf5c_o
 +
    two-to-one{13209}:  44.1.1.1/32 === 44.2.2.2/32
 +
If they are not up, check /var/log/syslog and restart ipsec on both servers.  Some times a server can get in a bad status if there is a mis-config.  Also it's worth noting that IPSEC is processed by iptables once it's decrypted, the <code>iptables -A INPUT -m policy --dir in --pol ipsec -j ACCEPT --src "$IPSECsrc" </code> line in the firewall allows all IPsec packets once decrypted to bypass the firewall.  This is able to prevent traffic between unencrypted services on the servers (e.g. mysql will not connect if the ipsec is down).
 +
 +
=== Configure Postfix ===
 +
 +
Postfix is installed to forward mail for root to a smtp host.
 +
 +
<code>apt-get install postfix mailutils</code>
 +
 +
This will run an installer with a curses interface and you must select '''Satallite System'''.  Check the '''System mail''' name is the hostname of the server, and the '''SMTP relay host''' is ''morty.keekles.org''. '''Root and postmaster mail''' should be ''rootmail@allstarlink.org''.
 +
 +
Should you need to reconfigure this use:
 +
 +
<code> dpkg-reconfigure postfix </code>
 +
 +
other aliases are setup in /etc/aliases.  You must run ''newaliases'' after this is updated for them to take effect.
 +
 +
= Verification =
 +
 +
It's important to verify the server provisiong before being put into production.
 +
 +
== Items to check ==
 +
* reboot the server/vm, do all services start properly?
 +
* Is the IP address configured on the server on eth0?
 +
* Is the hostname set?
 +
* Is it configured in DNS both forward and reverse?
 +
* Is the firewall active (try netcat on a non-permitted port)
 +
* IPSEC is active <code>ipsec status</code>?
 +
* Does Screen work in an xterm with scroll back?
 +
* Is the time set via ntp <code>ntptime</code> and is the timezone set to UTC?
 +
* Is fail2ban working?  Make a couple test connections and see if the IP is null routed <code>ip route show</code>
 +
 +
You may need to check your other services on this server now.
 +
 +
= Network Monitoring =
 +
It's time to hand off the server to the NMS team.  Please ensure SNMP is configured and an IPSEC tunnel is built to nms.allstarlink.org
 +
Logging will be sucked up by graylog.
 +
 +
Please ensure it's being watched in librenms by asking on the admin list or in the slack.
Retrieved from "http://wiki.pttlink.org/wiki/Special:MobileDiff/1005...10921"

Navigation menu