Technology Pavers

Many people think that IT departments are just the guys that fix your broken mice and are the guys responsible when your emails are slow. This is not the full picture though. The role of IT to an organization is not to be understated. IT is the team that every employee in your company depends on from the unpaid interns to the C-level execs. We (IT) are not just the guys you call when stuff breaks. We are the guys responsible for testing and adopting the latest technology and paving the roads that the company drives on. An IT organization can either enable exceptional contributors to do exceptional things, or it can tie the hands of your company’s talent and stifle innovation. The effect that a well oiled IT team can have on a company is paramount to a company’s success.

As an IT leader, I always closely follow the technology trends within the industry. Not all trends are a good fit for every company, and it is my responsibility to evaluate the trends and assess what the company can benefit from. External trends are not the only trends that I closely monitor, I also monitor internal trends as well. When innovative product developers explore the use of new technology, my job is to stay a step ahead and “pave the road” so that they can drive innovation without worrying about technology being an obstacle.

Whether your goals are as trivial as ensuring that your employees have the best equipment, or as complex as ensuring that your customers have access to your company’s digital presence 24x7x365 with the minimal latency, it is IT that makes it all happen. Exceptional IT leaders, and their teams, pave the way for exceptional employees to do exceptional things.

The Unprecedented Importance of Revisiting IT Precedents

Every IT leader would agree that a few of the top goals of any IT department are to provide reliable, secure, and sufficient services to meet the needs of the company that they support. Most IT leaders would also agree that yesterday’s solution is for yesterday’s needs. If today’s needs are different, then yesterday’s solutions are likely insufficient. This is clear to most people. The blind spot for some leaders is recognizing that solutions continuously evolve even if your needs remain static. Today’s unchanged solution to your unchanged needs might seem sufficient, but it might be hurting your company if it goes unchecked.

Whether you have worked with the company for 2 years or 20 years, it is important to revisit the precedents for solutions that have been set by you or others on a periodic basis. Each year, I add to my IT action plan to review at least one past large-scale solution with the same basic goals to achieve: better service, and better value.

Two years ago, I chose to revisit the company’s telecommunications infrastructure. As anticipated, I had the goals of providing better service and better value to the company and our customers. In pursuit of these goals, I met with our sales and support teams and asked about any pain points that they had and what an ideal solution would look like for them. I met with our finance team and compiled a complete TCO report for our existing telecommunications infrastructure. To provide better value, it was imperative to know the big picture of our total costs. Any new solution would need to provide compelling enough value and service for it to make sense. After a few months of research, and many SC led demo’s we found our solution. We tossed our very large footprint legacy on-premises PBX (including the PRI, and ACD, AES, SIP, etc.) in favor of software phones and a standards-based SIP solution with a managed cloud PBX. It has been two years since that project was completed and the our sales and support teams still tell me how happy they are with the new phone system. Not only did this project save the company $30K per year, but the sales team is enjoying mobility options that had never existed before. Our sales team can work from home and even take calls to their office line from the road.

The best part of the whole project was when I saw the smiles on our sales team’s faces a few weeks after the project was completed. The company’s needs had not changed since the last solution was implemented. But by contrasting the viable incumbent solution against leading modern solutions, I was able to sponsor and implement the change that unlocked unforeseen productivity gains and simultaneously improved the company’s bottom-line.

Creating a GlusterFS Cluster for VMs

The best GlusterFS layout that I have found to work with VMs is a distributed replicated cluster with shards enabled. I use laptops with 2 drives in each one. Let’s say we have 4 laptops with 2 drives on each one, we would do something like what I have listed below.

First, let’s create a partition on each disk. Use fdisk /dev/sda and fdisk /dev/sdb to create a partition. Now, each disk needs to be formatted as XFS. We format with the command.

mkfs.xfs -i size=512 /dev/sda1
mkfs.xfs -i size=512 /dev/sdb1
mkdir /mnt/disk1
mkdir /mnt/disk2

Now we can use blkid to list the UUID of sda1 and sdb1 so that we can add it to fstab. My fstab looks something like this (your UUIDs will be different. The allocsize is set to pre-allocate files 64MB at a time to limit fragmentation and improve performance. The noatime is to prevent the access time attribute being set every time the file is touched – this is to improve performance as well. The nofail is to prevent the system from not booting in the event of a disk failure.

#!/etc/fstab
UUID=3edc7ec8-303a-42c6-9937-16ef37068c72        /mnt/disk2    xfs     defaults,allocsize=64m,noatime,nofail  0 1
UUID=b8906693-27ba-466b-9c39-8066aa765d2e       /mnt/disk1      xfs     defaults,allocsize=64m,noatime,nofail  0 1

Now I did something funky with my fstab because I wanted to mount my bricks under the volname so that I could have different volumes on the same disks. So I added these lines to my fstab (my volname is “prod”).

#!/etc/fstab
/mnt/disk1/prod/brick1  /mnt/gluster/prod/brick1        none    bind    0 0
/mnt/disk2/prod/brick2  /mnt/gluster/prod/brick2        none    bind 0 0
mkdir -p /mnt/gluster/prod/brick1
mkdir -p /mnt/gluster/prod/brick2

mount /mnt/disk1
mount /mnt/disk2
mkdir -p /mnt/disk1/prod/brick1
mkdir -p /mnt/disk2/prod/brick2

Now we can mount everything else.

mount -a

Make sure that everything is mounted properly.

df -h /mnt/gluster/prod/brick1

Make sure that you see /dev/sda1 next to it. If not, just reboot and fstab will mount everything appropriately.

Now let’s create the gluster cluster.

gluster volume create prod replica 2 gluster1:/mnt/gluster/prod/brick1 gluster2:/mnt/gluster/prod/brick1 gluster3:/mnt/gluster/prod/brick1 gluster4:/mnt/gluster/prod/brick1 gluster1:/mnt/gluster/prod/brick2 gluster2:/mnt/gluster/prod/brick2 gluster3:/mnt/gluster/prod/brick2 gluster4:/mnt/gluster/prod/brick2

By specifying the order as we did, we ensure that server gluster1 and server gluster2 are paired up with each other, and gluster3 and gluster4 are paired up with each other. If we did both bricks on gluster1 successively, then we would be unable to sustain a failure of the gluster1 node. So we alternate servers. This also improves performance.

Now, the only thing left is to tune some parameters meant for VMs. For each parameter below, we will use a command like the following:

gluster volume set prod storage.fips-mode-rchecksum on

Here are my options:

storage.fips-mode-rchecksum: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
performance.strict-o-direct: on
server.event-threads: 3
client.event-threads: 8
performance.io-thread-count: 32
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: on
cluster.eager-lock: enable
performance.stat-prefetch: off
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
features.shard-block-size: 67108864
features.shard: on
server.allow-insecure: on
cluster.data-self-heal-algorithm: full
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet

Now we can fire up the gluster cluster with the following command.

gluster volume start prod

I pity the fool…

Recently, there seems to be an inordinate amount of bad luck surrounding my family. My wife passed out in February and crashed her car with the girls in it – thankfully everyone was ok – but it could have been tragic. As a result of that medical event, my wife can no longer drive for at least a year, perhaps longer. As you can imagine, this puts a huge hardship on the entire family. To make things more complicated, she continues to have episodes of passing out every once in a while, which makes it prudent for her not to carry the kids up or down stairs anymore. Back to the driving – not only can she not drive, but I have to take her for many doctor appointments while the doctors try to figure out what is wrong, and I also have had to re-arrange my work arrangements so that I can spend more time working from home so that I can drive everyone places. I am working longer hours to make up the time, doing more chores at home, and pretty much burning the candle at both ends. Where am I going with all of this? It’s easy for someone in a predicament like this to ask for pity. It’s easy to lose track of all of the good things in the world and focus on the current uptick in negative things. But that’s just not in my DNA. I don’t want pity, though help is always welcome. I come home every day from work and give the family a big hug. I’m happy that everyone is healthy, and that I get to see and enjoy their presence another day. So in the face of all of the recent bad luck bestowed on my family, I still feel like the luckiest guy around to have, and be a part of, such an amazing family.

Create a UEFI bootable ISO on Debian

First thing’s first… let’s get a quick overview of what’s need, what goes where, and what to expect.

In order to make an ISO bootable, you need an .img file. Many tutorials call this efiboot.img. It’s basically a FAT formatted file that contains a specific folder structure and a specially named executable that UEFI will run. The folder structure should have just one file in it located in /efi/boot/bootx64.efi. The bootx64.efi file is the boot code, so it can be any boot loader that generates it. I like Grub, so that’s what my instructions below do.

Now you may wonder – if you use Grub as the bootx64.efi, then where the heck does it get grub.cfg from? It’s pretty plain and simple – it’s in /boot/grub/grub.cfg just as it would be in a regular filesystem. Except, /boot/grub/grub.cfg lives outside the .img file — it lives along-side it, really.

Ok, now that that’s out of the way, let’s get started. You need some packages first.

sudo apt update
sudo apt install grub-efi-amd64 grub-efi-amd64-bin grub-imageboot grub-legacy grub-pc-bin grub-pc

Now let’s create a rescue image, which will do the heavy lifting for us. Then we will mount it to a temp folder.

grub-mkrescue -o grub.iso
mkdir /tmp/grubcd
mount -o loop grub.iso /tmp/grubcd

Now let’s create the folder where we want our ISO to live. Then we will copy the contents of grub.iso to that folder.

mkdir myiso
cp -avR /tmp/grubcd/* myiso/

Next, let’s create a grub.cfg file so that you can tell it where and what to load. You will want to edit this and not leave it blank.

touch myiso/boot/grub/grub.cfg

Lastly, copy your vmlinuz and initrd files to myiso, along with any other files you need/want. Then you can create your iso with something like xorriso.

cd myiso
xorriso -volid "CDROM" -as mkisofs -isohybrid-mbr isohdpfx.bin -b isolinux.bin -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e grub.img -isohybrid-gpt-basdat -o ../mycd.iso ./

Arch on Chromebook

Apps…

pacman -Sy
pacman -S chromium xorg-server connman enlightenment rxvt-unicode autocutsel; # Note: Choose the noto fonts when prompted
systemctl enable connman

Touchpad support:
/usr/share/X11/xorg.conf.d/70-synaptics.conf

# Example xorg.conf.d snippet that assigns the touchpad driver
# to all touchpads. See xorg.conf.d(5) for more information on
# InputClass.
# DO NOT EDIT THIS FILE, your distribution will likely overwrite
# it when updating. Copy (and rename) this file into
# /etc/X11/xorg.conf.d first.
# Additional options may be added in the form of
#   Option "OptionName" "value"
#
Section "InputClass"
        Identifier "touchpad catchall"
        Driver "synaptics"
        MatchIsTouchpad "on"
# This option is recommend on all Linux systems using evdev, but cannot be
# enabled by default. See the following link for details:
# http://who-t.blogspot.com/2010/11/how-to-ignore-configuration-errors.html
#       MatchDevicePath "/dev/input/event*"
EndSection

Section "InputClass"
        Identifier "touchpad ignore duplicates"
        MatchIsTouchpad "on"
        MatchOS "Linux"
        MatchDevicePath "/dev/input/mouse*"
        Option "Ignore" "on"
EndSection

# This option enables the bottom right corner to be a right button on clickpads
# and the right and middle top areas to be right / middle buttons on clickpads
# with a top button area.
# This option is only interpreted by clickpads.
Section "InputClass"
        Identifier "Default clickpad buttons"
        MatchDriver "synaptics"
        Option "TapButton1" "1"
        Option "TapButton2" "3"
        Option "TapButton3" "2"
        Option "SoftButtonAreas" "50% 0 82% 0 0 0 0 0"
        Option "SecondarySoftButtonAreas" "58% 0 0 15% 42% 58% 0 15%"
EndSection

# This option disables software buttons on Apple touchpads.
# This option is only interpreted by clickpads.
Section "InputClass"
        Identifier "Disable clickpad buttons on Apple touchpads"
        MatchProduct "Apple|bcm5974"
        MatchDriver "synaptics"
        Option "SoftButtonAreas" "0 0 0 0 0 0 0 0"
EndSection

xterm / rxvt customizations
~/.Xdefaults

! Perl extension config
URxvt.perl-ext-common: default,selection-to-clipboard
URxvt.perl-ext: tabbed
! Any scripts placed here will override global ones with the same name

!-- Xft settings -- !
!Xft.dpi:        96
!Xft.antialias:  true
!Xft.rgba:       rgb
!Xft.hinting:    true
!Xft.hintstyle:  hintfull

! Tabbed extension configuration
URxvt.tabbed.tabbar-fg: 8
URxvt.tabbed.tabbar-bg: 0
URxvt.tabbed.tab-fg:    15
URxvt.tabbed.tab-bg:    8
URxvt.tabbed.new-button: false

URxvt*depth: 32
URxvt*background: rgba:0000/0000/0200/c800
URxvt*font: xft:NotoSansMono-Medium:size=18:antialias=true
URxvt*foreground:white

/etc/sudoers.d/xorg

%sudo ALL=(ALL)NOPASSWD:/usr/bin/Xorg
%sudo ALL=(ALL)NOPASSWD:/usr/local/bin/Xorg

Uncomment the line in /etc/sudoers to look like this:

%wheel ALL=(ALL) ALL

Add user myuser (change to the name of your user) to wheel.

usermod -aG wheel myuser

Create a file /usr/bin/startx-custom

sudo /usr/bin/Xorg &
sleep 2
export DISPLAY=:0
/usr/bin/autocutsel -fork
if [ `pidof Xorg` ]; then
        /usr/bin/enlightenment_start && killall -s KILL Xorg
else
        echo "No Xorg running... not starting E17"
fi

Make it executable.

chmod a+x /usr/bin/startx-custom

For a high resolution screen like on the Samsung Chromebook Plus, you may want to scale Chromium. To do this, add the following.
~/.config/chromium-flags.conf

--force-device-scale-factor=1.7

If DNS is not working right, adjust /etc/nsswitch.conf to look like this:

hosts: files mymachines myhostname dns resolve [!UNAVAIL=return] dns

Fail-To-Ban (Lite) – EdgeRouter

Here’s how to create a fail-to-ban type of functionality on an EdgeRouter completely using BASH, without installing any 3rd party packages. We are going to create a single script and add a scheduled job to run it. That’s all there is to it.

Step 1
Run the following

vi /config/scripts/fail-to-ban

Now we need to turn off auto indent before copying and pasting the script below. Type the following:

:set noai
i

Now paste the following script into the file:

#!/bin/bash

PATH=$PATH:/usr/bin:/bin:/sbin:/usr/sbin

ATTEMPTS=10;    # NUMBER OF ATTEMPTS IN A GIVEN INTERVAL
INTERVAL=600;   # INTERVAL (IN SECONDS) TO WATCH FOR FAILED ATTEMPTS - HISTORICALLY FROM CURRENT TIME
PERMBAN=100;    # AFTER THIS NUM OF FAILED ATTEMPTS, BAN UNTIL LOG ROTATES
BLOCKSECS=3600; # AFTER THIS TIME (IN SECONDS), UNBLOCK A BLOCKED IP
BLOCKED_ALREADY=""
BLOCKED_NOW=""
SKIPPED=""
EXPIRED_BLOCK=""
NOW=`date '+%s'`

isip() {
	ISIP=$1
	if [ $(echo $IP | sed 's/[^.]//g' | awk '{print length; }' 2> /dev/null) -eq 3 ]; then
		ISIP=1
	fi
}

fail2ban() {
        # echo failing $IP with count $COUNT and lastcount $LASTCOUNT
        IP=$IP
        EXISTS=`nice -19 iptables -n -L | grep $IP | wc -l`
        IS_LOCAL=`echo $IP | grep -E '^10\.|192\.168|127\.' | wc -l`
        if [ $EXISTS -gt 0 ]; then
		BLOCKED_ALREADY+=",$IP:$COUNT"
                # echo "IP $IP is already blocked"
        elif [ $IS_LOCAL -eq 1 ]; then
		SKIPPED+=",$IP:$COUNT"
                # echo "IP is local IP.  Not blocking"
        else
		if [ ! "$IP" == "" ]; then
	                # echo "Blocking IP $IP after $COUNT abuses."
	                BLOCKED_NOW+=",$IP:$COUNT"
	                iptables -I INPUT 1 -j DROP -s $IP
	                echo "`date`:$IP:$NEWCOUNT:$COUNT:BLOCKED" >> /tmp/banned.log
		fi
        fi
}

updateList() {
        NOW=`date '+%s'`
        sed -i /tmp/ip-list.log -e "s/:"$IP":"$LASTCOUNT".*$/:"$IP":"$COUNT":"$NOW"/"
}

updateTime() {
	NOW=`date '+%s'`
	sed -i /tmp/ip-list.log -e "s/:"$IP":"$LASTCOUNT".*$/:"$IP":"$LASTCOUNT":"$NOW"/"
}


showList() {
	LIST="$2"
	DESCRIPTION="$1"
	if [ ! "$LIST" == "" ]; then  
        	echo "$DESCRIPTION"
        	for i in `echo "$LIST"`                                                                       
        	do                                                                                                   
        	        BIP=$(echo $i | sed -e 's/:.*$//')                                                           
        	        BCOUNT=$(echo $i | sed -e 's/^.*://')                                                        
			if [ ! "$BIP" == "" ]; then
		                echo $BIP $BCOUNT                                                                            
			fi
        	done
	fi	
}

checkExpired() {
	BLOCKED=$(nice -19 iptables -L INPUT -n | grep "^DROP" | sed -e 's/^.*--  //' | sed -e 's/ .*$//')
	for i in `grep -e "$BLOCKED" /tmp/ip-list.log`                                                                                                                       
	do                                                                                                                                                                   
	        IP=`echo $i | cut -d':' -f2`                                                                                                                                 
	        isip $IP                                                                                                                                                     
	        COUNT=`echo $i | cut -d':' -f3`                                                                                                                              
	        LASTACTION=`echo $i | cut -d':' -f4`                                                                                                                         
	        if [ $((NOW-LASTACTION)) -gt $BLOCKSECS ] && [ ! "$IP" == "" ] && [ $ISIP -eq 1 ] && [ $COUNT -lt $PERMBAN ]; then                                           
	                LINE=`nice -19 iptables -L -n --line-numbers | grep "$IP" | cut -d' ' -f1`                                                                           
	                if [ ! "$LINE" == "" ]; then                                                                                                                         
	                        echo "Removing block on $IP"                                                                                                                 
				echo "$(date):$IP:UNBLOCKED" >> /tmp/banned.log
	                        # EXPIRED_BLOCK+=",$IP"                                                                                                                      
	                        echo iptables -D INPUT $LINE                                                                                                                 
	                        iptables -D INPUT $LINE                                                                                                                      
	                fi                                                                                                                                                   
	        fi                                                                                                                                                           
	done                                   
}


if [ ! -f /tmp/ip-list.log ]; then
        touch /tmp/ip-list.log
fi

# CLEANUP - KEEP ONLY HACKERS FROM TODAY
echo -n "" > /tmp/ip-list.new
IFS="
"
for i in `grep "^$(date +%Y%m%d):" /tmp/ip-list.log`
do
	if [ ! "$i" == "" ]; then
		echo $i >> /tmp/ip-list.new
	fi
done
mv /tmp/ip-list.new /tmp/ip-list.log

# Do some checking to see if the logs actually changed
if [ -f /tmp/this-run ]; then
	mv /tmp/this-run /tmp/last-run
else
	touch /tmp/last-run
fi
ls -1e /var/log/auth.log > /tmp/this-run
CHANGE=$(diff /tmp/last-run /tmp/this-run | wc -l)
if [ $CHANGE -eq 0 ]; then
	echo "No change since last run"
	checkExpired
	exit
fi

IPLIST=`nice -19 grep failure /var/log/auth.log | grep rhost | sed -e 's/^.*rhost=//' | sed -e 's/ .*$//' | sort | uniq -c | sed -e 's/^ *//' | sed -e 's/ /:/' | grep -E ":[
0-9.]*$" | sed -e "s/^\(.*\)$/$(date +%Y%m%d):\1/"`

for i in `echo "$IPLIST"`
do
	#echo $i
        COUNT=`echo $i | cut -d':' -f2`
        IP=`echo $i | cut -d':' -f3`
	DATE=`echo $i | cut -d':' -f1`
	isip $IP
        LASTCOUNT=`cat /tmp/ip-list.log | grep ":$IP:" | cut -d':' -f3`
        ELAPSED=`cat /tmp/ip-list.log | grep ":$IP:" | cut -d':' -f4 | sed -e 's/\n//g'`
	ELAPSED=$((NOW-ELAPSED))
        if [ "$COUNT" == "" ]; then
                COUNT=0
        fi
        if [ "$LASTCOUNT" == "" ]; then
                LASTCOUNT=0
        fi
        NEWCOUNT=$((COUNT-LASTCOUNT))
        if [ ! "$LASTCOUNT" == "" ] && [ $LASTCOUNT -eq 0 ] && [ $ISIP -eq 1 ]; then
                echo "$DATE:$IP:$COUNT:$NOW" >> /tmp/ip-list.log
               # echo "Adding $IP to the IP tracking log with count $COUNT"
        fi
	# echo "IP:$IP NEWCOUNT:$NEWCOUNT LASTCOUNT:$LASTCOUNT COUNT:$COUNT ELAPSED:$ELAPSED ISIP:$ISIP ATTEMPTS:$ATTEMPTS INTERVAL:$INTERVAL"
	
        if [ $NEWCOUNT -ge $ATTEMPTS ] && [ $ISIP -eq 1 ] && ( [ $ELAPSED -le $INTERVAL ]  ||  [ $COUNT -gt $PERMBAN ] ); then
                if [ $LASTCOUNT -ne 0 ]; then
			echo "Updating IP:$IP with NEWCOUNT:$NEWCOUNT ATTEMPTS:$ATTEMPTS ELAPSED:$ELAPSED INTERVAL:$INTERVAL ISIP:$ISIP"
                        updateList

                fi
                fail2ban
	elif [ $NEWCOUNT -ge $ATTEMPTS ] && [ $ISIP -eq 1 ]; then
		echo "Updating the timestamp for IP $IP; +$NEWCOUNT since last update"
		updateTime
        fi
done

checkExpired

IFS=","

showList "Blocked | Attempts" "$BLOCKED_ALREADY"
showList "Newly Blocked | Attempts" "$BLOCKED_NOW"
showList "Skipped | Attempts" "$SKIPPED"
showList "Expired" "$EXPIRED_BLOCK"

Now you need to exit and save your changes.

esc
:wq

Now let’s make the script executable:

chmod a+x /config/scripts/fail-to-ban

And lastly, we need to add a job to run this periodically and also disable hostname lookups in SSH to make the script’s IP validation work properly and remove a possible DoS vector.

configure
set system task-scheduler task failtoban executable path /config/scripts/fail-to-ban
set system task-scheduler task failtoban interval 1m
set service ssh disable-host-validation
commit
save

Done! Now your EdgeRouter should be protected against brute force login attacks.

How to replace a brick in GlusterFS

Scenario 1: Your data drive fails in your machine and you really don’t want to go through the process of reconfiguring a new system just to create and join a new brick.

Scenario 2: You proactively replace your hard drive and you keep the OS in tact and just want to use the new drive.

The solution to either scenario is quite simple.

Let’s stop cron just in case we have any watchdog processes running.

service cron stop

First thing we need to do is tell GlusterFS that we want to take the brick offline. We do that with this command.

gluster volume reset-brick your-volume your-hostname-or-ip:/path-to-failed-brick

Then we can unmount the old volume and swap out a new disk. As long as our config remains the same, we will later use the reset-brick command to get all of the data back from the redundant copy.

Now, you need to create a partition and then format the new drive with the following. Let’s assume that your new drive/partition is /dev/sdb1 and that it’s an SSD so that you want to use f2fs.

mkfs.f2fs /dev/sdb1

Now, you need to update /etc/fstab so that it automatically mounts. Let’s say your mount point is /mnt/gluster. You will want to use something like this:

echo "UUID=`blkid | grep /dev/sdb1 | cut -d'\"' -f2` /mnt/gluster        f2fs     defaults,noatime,nofail    0 1" >> /etc/fstab

Note: You should comment out the old entry in your fstab file.

Next, you will want to mount your new partition.

mount -a

Now let’s create the new brick. This must be a different name than the old one!! Assuming that your old brick is /mnt/gluster/brick, we could simply do.

mkdir -p /mnt/gluster/brick

Now let’s use the gluster volume reset-brick command to do the heavy lifting for us.

gluster volume reset-brick your-volume your-hostname-or-ip:/path-to-failed-brick your-hostname-or-ip:/path-to-failed-brick commit force

If you stopped cron, start it back up now.

service cron start

That’s it.

Now you can check on the progress of the re-population of data by using the heal info command.

Hope this helps!

Simple Fail2Ban Alternative for FreePBX + SSH

I created a script to help secure Asterisk servers: fail-to-ban. It runs once per minute and will block an IP that tries, unsuccessfully, to authenticate more than a certain number of tries within a certain interval. It keeps track of each block and will automatically unblock an IP after a given amount of time elapses. All of these variables are easily set/tuned at the beginning of the script. It’s basic, easy to install, and effective. The installation is just two extremely easy steps.

  • Create the file /scripts/fail-to-ban.
  • Add a cron job entry to run it.
  • /scripts/fail-to-ban

    #!/bin/bash
    
    PATH=$PATH:/usr/bin:/bin:/sbin:/usr/sbin
    
    ATTEMPTS=10;    # NUMBER OF ATTEMPTS IN A GIVEN INTERVAL
    INTERVAL=300;   # INTERVAL (IN SECONDS) TO WATCH FOR FAILED ATTEMPTS - HISTORICALLY FROM CURRENT TIME
    PERMBAN=100;    # AFTER THIS NUM OF FAILED ATTEMPTS, BAN UNTIL LOG ROTATES
    BLOCKSECS=3600;   # AFTER THIS TIME (IN SECONDS), UNBLOCK A BLOCKED IP
    BLOCKED_ALREADY=""
    BLOCKED_NOW=""
    SKIPPED=""
    EXPIRED_BLOCK=""
    NOW=`date '+%s'`
    
    isip() {
            ISIP=$1
            if [ $(echo $IP | sed 's/[^.]//g' | awk '{print length; }' 2> /dev/null) -eq 3 ]; then
                    ISIP=1
            fi
    }
    
    fail2ban() {
            # echo failing $IP with count $COUNT and lastcount $LASTCOUNT
            IP=$IP
            EXISTS=`iptables -n -L | grep $IP | wc -l`
            IS_LOCAL=`echo $IP | grep -E '^10\.|192\.168|127\.' | wc -l`
            if [ $EXISTS -gt 0 ]; then
                    BLOCKED_ALREADY+=",$IP:$COUNT"
                    # echo "IP $IP is already blocked"
            elif [ $IS_LOCAL -eq 1 ]; then
                    SKIPPED+=",$IP:$COUNT"
                    # echo "IP is local IP.  Not blocking"
            else
                    if [ ! "$IP" == "" ]; then
                            # echo "Blocking IP $IP after $COUNT abuses."
                            BLOCKED_NOW+=",$IP:$COUNT"
                            iptables -I INPUT 1 -j DROP -s $IP
                            echo "`date`:$IP:$NEWCOUNT:$COUNT" >> /tmp/banned.log
                    fi
            fi
    }
    
    updateList() {
            NOW=`date '+%s'`
            sed -i /tmp/ip-list.log -e "s/"$IP":"$LASTCOUNT".*$/"$IP":"$COUNT":"$NOW"/"
    }
    
    
    showList() {
            LIST="$2"
            DESCRIPTION="$1"
            if [ ! "$LIST" == "" ]; then  
                    echo "$DESCRIPTION"
                    for i in `echo "$LIST"`                                                                       
                    do                                                                                                   
                            BIP=$(echo $i | sed -e 's/:.*$//')                                                           
                            BCOUNT=$(echo $i | sed -e 's/^.*://')                                                        
                            if [ ! "$BIP" == "" ]; then
                                    echo $BIP $BCOUNT                                                                            
                            fi
                    done
            fi
    }
    
    checkExpired() {
            BLOCKED=$(iptables -L INPUT -n | grep "^DROP" | sed -e 's/^.*--  //' | sed -e 's/ .*$//')
            for i in `grep -e "$BLOCKED" /tmp/ip-list.log`                                                                                                                                              
            do                                                                                                                                                                           
                    IP=`echo $i | cut -d':' -f1`                                                                                                                                         
                    isip $IP                                                                                                                                                             
                    COUNT=`echo $i | cut -d':' -f2`                                                                                                                                      
                    LASTACTION=`echo $i | cut -d':' -f3`                                                                                                                                 
                                                                                                                                                                                         
                    if [ $((NOW-LASTACTION)) -gt $BLOCKSECS ] && [ ! "$IP" == "" ] && [ $ISIP -eq 1 ] && [ $COUNT -lt $PERMBAN ]; then                                                   
                            LINE=`iptables -L -n --line-numbers | grep "$IP" | cut -d' ' -f1`                                                                                            
                            if [ ! "$LINE" == "" ]; then                                                                                                                                 
                                    echo "Removing block on $IP"                                                                                                                         
                                    # EXPIRED_BLOCK+=",$IP"                                                                                                                              
                                    echo iptables -D INPUT $LINE                                                                                                                         
                                    iptables -D INPUT $LINE                                                                                                                              
                            fi                                                                                                                                                           
                    fi                                                                                                                                                                   
            done                                   
    }
    
    
    if [ ! -f /tmp/ip-list.log ]; then
            touch /tmp/ip-list.log
    fi
    
    # Do some checking to see if the logs actually changed
    if [ -f /tmp/this-run ]; then
            mv /tmp/this-run /tmp/last-run
    else
            touch /tmp/last-run
    fi
    ls -al /var/log/asterisk/security > /tmp/this-run
    CHANGE=$(diff /tmp/last-run /tmp/this-run | wc -l)
    if [ $CHANGE -eq 0 ]; then
            echo "No change since last run"
            checkExpired
            exit
    fi
    
    IPLIST=$(/bin/grep -E "InvalidAccount|ChallengeResponseFailed" /var/log/asterisk/security | sed -e 's/^.*RemoteAddress="IPV4\/UDP\/\([0-9.]*\)\/.*$/\1/' | sort | uniq -c | sed -e 's/^ *//' | sed -e 's/ /:/')
    
    for i in `echo "$IPLIST"`
    do
            #echo $i
            COUNT=`echo $i | cut -d':' -f1`
            IP=`echo $i | cut -d':' -f2`
            isip $IP
            LASTCOUNT=`cat /tmp/ip-list.log | grep "^$IP:" | cut -d':' -f2`
            ELAPSED=`cat /tmp/ip-list.log | grep "^$IP:" | cut -d':' -f3 | sed -e 's/\n//g'`
            ELAPSED=$((NOW-ELAPSED))
            if [ "$COUNT" == "" ]; then
                    COUNT=0
            fi
            if [ "$LASTCOUNT" == "" ]; then
                    LASTCOUNT=0
            fi
            NEWCOUNT=$((COUNT-LASTCOUNT))
            if [ ! "$LASTCOUNT" == "" ] && [ $LASTCOUNT -eq 0 ] && [ $ISIP -eq 1 ]; then
                    echo "$IP:$COUNT:$NOW" >> /tmp/ip-list.log
                   # echo "Adding $IP to the IP tracking log with count $COUNT"
            fi
            if [ $NEWCOUNT -ge $ATTEMPTS ] && [ $ISIP -eq 1 ] && ( [ $ELAPSED -le $INTERVAL ]  ||  [ $COUNT -gt $PERMBAN ] ); then
                    if [ $LASTCOUNT -ne 0 ]; then
                            # echo "Updating IP:$IP with NEWCOUNT:$NEWCOUNT ATTEMPTS:$ATTEMPTS ELAPSED:$ELAPSED INTERVAL:$INTERVAL ISIP:$ISIP"
                            updateList
    
                    fi
                    fail2ban
            fi
    done
    
    checkExpired
    
    IFS=","
    
    showList "Blocked | Attempts" "$BLOCKED_ALREADY"
    showList "Newly Blocked | Attempts" "$BLOCKED_NOW"
    showList "Skipped | Attempts" "$SKIPPED"
    showList "Expired" "$EXPIRED_BLOCK"
    

    /scripts/crontab

    * * * * * /scripts/fail-to-ban > /dev/null 2>&1 &
    

    After creating these files, simply run the following commands:

    chmod a+x /scripts/fail-to-ban
    crontab /scripts/crontab
    

    That’s really all you need to do!