IMPORTANT: Read the setup guide first.
http://aro.wiki/how-to-setup-an-arionum-masternode/

This is a basic overview of how to secure your Arionum nodes. This is not a comprehensive guide. But, it should be enough to keep your nodes safe. ESPECIALLY your masternodes.

NOTE: Working on this right now 2018/08/12… More will be added as I progress and monitor…

Linux is Easy

  • Almost everything on a linux system is a simple text file.
  • There are many flavors of linux. I prefer Ubuntu. Most commands on Ubuntu also work on Debian systems. Fedora/RedHat/CentOS are all similar to each other but have different setups than Ubuntu/Debian.
  • My guides will talk about Ubuntu only.
  • Learn how to use nano text editor
    • $ nano filename
    • To save file: CTRL+X, then Y for Yes, then ENTER
  • Most config files for services are located in /etc/* and can be edited with nano text editor.
  • Most log files are located in /var/log/*
  • Usually website code/files are located in /var/www/yourdomain.com/*
  • Linux has users with home folders. Each home folder has config files for that user that start with .
    • /root/ is the home folder for root user.
    • Any other users you create will be in /home/yourusername/
  • Most common commands have a help page that tell you all the options available: type “help cd”
  • Commands you should learn: cd, rm, cp, mv, top, ls -alih, apt-get

Lock Down SSH Access

  • If your hosting provider has SSH key feature during setup do it. Backup your key and keep it safe. What you do is setup a local private and public key on your own computer. Then you copy the public key to the server.Then you can SSH to your server from your computer with your key. Checkout these tutorials…
  • The following is a configuration for your SSH access usually /etc/ssh/sshd_config
    • Set …
      • “PasswordAuthentication no” since we are going to use SSH keys.
      • PermitEmptyPasswords no
      • PubkeyAuthentication yes
      • AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2
    • Change your SSH port. Something like “Port 22987” or any number between like 2200 and 65000. If you setup a firewall you need to add this “Custom TCP Port 22987” as Inbound Allowed from anywhere.
  • When you make changes to SSH config and restart ssh service. Do NOT logout!! Open a new session and make sure you can still connect otherwise you will have to go into console in hosting provider and fix it. You could also completely lock yourself out. If you dont close your other session you can fix it before you lock yourself out.
nano /etc/ssh/sshd_config;
# look for setting above and change them; 
# remove any # in front off line to enable the setting
# to save CTRL+X, Y, ENTER

service ssh restart;

Setup fail2ban

  • Setup fail2ban service.
  • This will monitor your /var/log/auth.log for brute force attempts by bots/hackers and will auto block them if they try too many times.
  • If you change your SSH port, you will need to change the word “port = ssh” in the fail2ban config to your new port number “port = 22789” or whatever your port is. I will provide instructions on this below.
  • Try this guide for more info: https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-14-04
sudo apt-get install fail2ban;
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local;

sudo nano /etc/fail2ban/jail.local;
# in config look for [ssh*] sections 
# replace "port = ssh" with "port = 23456" 
# or whatever your SSH port is
# to save CTRL+X, Y, ENTER

sudo service fail2ban restart;

Setup Network Firewall

  • This will be a firewall provided by your hosting provider and occurs at the networking layer BEFORE it reaches your node. Your hosting provider should provide a GUI you can use to make firewall rules. When you locate bad actors you can permanently ban/block them here.
  • Your masternode server should only allow TCP ports 80 and TCP 22 unless you changed your SSH port. If you ever enable HTTPS in the future you will want to allow TCP 443 as well but not required in this tutorial. Block all other inbound access. You can leave outbound access open.
  • Your wallet server should only allow TCP 22 unless you changed your SSH port. Block all other inbound access. You can leave outbound access open. I wouldn’t leave a wallet server online for too long. Shut it down. Only boot it up when you need it. Also you can run the wallet cli on any computer that has PHP cli. Installed. I am a developer. I already had it installed. I use it on my local computer as needed then put my wallets in a secure place when done.

Setup Software Firewall

Configure Automatic Security Updates

Setup Swap Space

  • If your node appears to be hanging and not keeping up you could have a memory issue. Easy solution is to upgrade server to have more RAM.
  • There is another bandaid you could do that gives you more RAM but the RAM is stored on disk which is SLOWER than real RAM. Its called SWAP space.
  • I suggest you setup swap space anyways to prevent OOM errors in your syslog.
  • If you have 4GB of RAM setup 2GB of SWAP space
# check log for OOM errors ...
grep "Out of memory" /var/log/syslog;
# kernel: Out of memory: Kill process 5573 (php-fpm7.2) ...

# Add swap space ...
swapon --show; # if blank you have none
free -h; # shows you memory
df -h; # shows disk space available
fallocate -l 2G /swapfile; # change 2G to 1G or other as you wish
ls -lh /swapfile;
chmod 600 /swapfile;
mkswap /swapfile;
swapon /swapfile;
swapon --show;
cp /etc/fstab /etc/fstab.bak;
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab;

# adjust swappiness
cat /proc/sys/vm/swappiness; # says 60?
sysctl vm.swappiness=10; 

nano /etc/sysctl.conf;
# this will open text editor, add to bottom...
vm.swappiness=10
# then CTRL+X, then Y, then ENTER to save.

# reboot server
reboot now

Optimize and Secure Mysql

PROBLEM: Database … “Too many connections”
SOLUTION: /etc/mysql/my.cnf  => max_connections
You may want to increase setting from 150 to 1000 then restart mysql.

# MANUALLY:
nano /etc/mysql/my.cnf;
sudo service mysql restart;

# WITH ARONODE SCRIPT:
bash aronode mainnet mysql get max_connections; # to see what it is
bash aronode mainnet mysql set max_connections 1000; # to set it
bash aronode mainnet retstart; # to load it

PROBLEM: /var/log/syslog showing “mysqld[1415]: 2018-08-17 20:47:57 1401123456769184 [Warning] Aborted connection to db: user: host: (Got timeout reading communication packets)”
SOLUTION: /etc/mysql/my.cnf  => max_allowed_packet

# WITH ARONODE SCRIPT:
bash aronode mainnet mysql get max_allowed_packet; # to see what it is
bash aronode mainnet mysql set max_allowed_packet 100M; # to set it
bash aronode mainnet restart; # to load it

More Info: https://dba.stackexchange.com/questions/19135/mysql-error-reading-communication-packets

Optimize and Secure Nginx

Block Bad Bots. They are everywhere. They are relentless. They will hit your server repeatedly looking for holes, gaps, vulnerabilities. If your server keeps going down this could be one reason.

Install and configure https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker

Its an automated bad bot blacklisting tool. The only manual thing you need to add to the install instructions is whitelisting all the main arionum peer IP addresses and/or *arionum.com.

sudo wget https://raw.githubusercontent.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/master/install-ngxblocker -O /usr/local/sbin/install-ngxblocker;
sudo chmod +x /usr/local/sbin/install-ngxblocker;
cd /usr/local/sbin;
sudo ./install-ngxblocker;
cd /usr/local/sbin/;
sudo ./install-ngxblocker -x;
sudo chmod +x /usr/local/sbin/setup-ngxblocker;
sudo chmod +x /usr/local/sbin/update-ngxblocker;
cd /usr/local/sbin/;
sudo ./setup-ngxblocker;
cd /usr/local/sbin/;
sudo ./setup-ngxblocker -x;
sudo nginx -t && sudo nginx -s reload;

sudo crontab -e;
# this will open crontab in nano; add the following line to bottom
00 */8 * * * sudo /usr/local/sbin/update-ngxblocker -n
# then CTRL+X, then Y, then ENTER 

nano /etc/nginx/bots.d/whitelist-domains.conf;
# this will open config in nano; add the following line (if not exists)
"~*arionum.com"     0;
# to save CTRL+X, then Y, then ENTER
 
nano /etc/nginx/bots.d/whitelist-ips.conf;
# this will open config in nano; add the following line (if not exists)
# some of the main arionum peers; NO, we dont need them all;
94.156.144.141  0;
185.203.119.89  0;
185.203.116.134 0;
185.203.119.88  0;
185.206.146.105 0;
212.73.150.66   0;
185.205.210.84  0;
85.217.170.77   0;
212.73.150.85   0;
# to save CTRL+X, then Y, then ENTER

nginx -t; # make sure it says "syntax is ok" 
bash aronode mainnet restart;

Other Interesting Configs:
2018/08/21: fastcgi_buffers 8 4k;
I added the following config to the .vhost file on github.
See https://gist.github.com/magnetikonline/11312172
See https://github.com/KyleFromOhio/arionum-scripts/blob/master/aronode.vhost

Optimize and Secure PHP

Problem:

# grep WARNING /var/log/php7.2-fpm.log;
[16-Aug-2018 21:05:50] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
[17-Aug-2018 00:59:41] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.
min/max_spare_servers), spawning X children, there are X idle, and X total children

Solution:

  1. Check RAM used per PHP script with ps command in box below.
  2. On one of my boxes i was seeing lots of… ~6.56MB round up to 7MB
  3. Masternodes: If server has 4GB of RAM; leave about 25% headroom, so:
    1. TESTED 2018/08/16: Appears stable
    2. pm.max_children = 4000MB * 75% = 3000MB / 7MB avg php request = 428
    3. EXAMPLE CONFIG: $ nano /etc/php/7.2/fpm/pool.d/www.conf;
      pm.max_children = 400 (rounded down for easy math)
      pm.start_servers = 200 (about 20-40% of max_children; increase if errors persist)
      pm.min_spare_servers = 100  (50-100% of start)
      pm.max_spare_servers = 200  (higher than min_spare_servers)
      pm.max_requests = 1000 (uncomment this out to stop memory leaks)
  4. Regular Nodes: If server has 2GB of RAM; leave some 30% headroom, so:
    1. 2018/08/16: Have not tested these 2GB Nodes yet. Half ram = half settings is theory…
    2. pm.max_children = 2000MB * 75% = 1500MB / 7MB = 214
    3. EXAMPLE CONFIG: $ nano /etc/php/7.2/fpm/pool.d/www.conf;
      pm.max_children = 200 (rounded down)
      pm.start_servers = 100 (about 20-40% of max_children; increase if errors persist)
      pm.min_spare_servers = 50  (50-100% of start)
      pm.max_spare_servers = 100  (higher than min_spare_servers)
      pm.max_requests = 500 (uncomment this out to stop memory leaks)
  5. $ nano /etc/php/7.2/fpm/pool.d/www.conf;
    1. This will open a text editor scroll down to find the settings, change them, then CTRL+X, Y, ENTER to save.
  6. Restart the services to reload the new settings $ bash aronode mainnet restart;
grep WARNING /var/log/php7.2-fpm.log; 
ps -eo size,pid,user,command --sort -size | awk '{ hr=$1/1024 ; printf("%13.2f Mb ",hr) } { for ( x=4 ; x<=NF ; x++ ) { printf("%s ",$x) } print "" }' | grep php-fpm
# should see some items like ...
# 30 Mb php-fpm: pool www
# 6.56 Mb php-fpm: pool www
# what are most of them listed ? probably 6.56 MB ?

nano /etc/php/7.2/fpm/pool.d/www.conf;
# this will open test file; scroll down and edit settings
# try my settings above OR do your own math
# to save CTRL+X, then Y, then ENTER

bash aronode mainnet restart;

Backups and Snapshots

Your hosting provider should offer a snapshot/backup service. I’d recommend at least once per week auto-backup. Once you have it configured and optimized and secured not much will change. You can run a manually snapshot and just save it in case you need to restore all your work back to day 1.

Passwords, 2FA Logins, and Authentication

  • Whenever and wherever possible use 2FA Logins. If you hosting provider has it. Set it up. If the exchanges you are on have it. Set it up. If your email has it. Set it up. NEVER use SMS as verification. Hackers can spoof SMS. Use Google Authenticator or Authy App.
  • People’s understanding of password strength math is terrible. Which password is better? FsR4S32SgH1 or HappyBlackPandaBear … the second one. Here is why. Read this funny comic https://xkcd.com/936/ Use four random memorable words with a capital letter. If the system requires numbers and funny characters too then do something like this: HappyBlackPandaBear69! Its FAR more secure and easy to manage than stupid random characters. Now go tell everyone else so people stop using these ridiculous password rules on their systems.

Leave a Reply

Your email address will not be published. Required fields are marked *