Why do we use Azure Virtual WAN instead of Azure Virtual Network Gateway?

Azure Virtual Network Gateway won't allow us to connect to the internet through the VPN connection.

How to setup Azure Virtual WAN for VPN?

The following are the major steps included in setting up Virtual WAN.

✅Create Virtual WAN

✅Create Virtual Hub

✅Setup Azure Firewall

✅Create Azure Firewall Policy

✅Associate Virtual Hub with Firewall Policy

✅Update Routing Intent & Routing Policies

✅Update Route Tables

✅Download VPN Configuration

✅Modify VPN Configuration

✅VPN Client Setup - Apple macOS

✅VPN Client Setup - Microsoft Windows

Create Virtual WAN

Login to Azure Portal and navigate to Azure Virtual WAN and click on create. Choose the Resource Group and Region and provide a Name. Click on Review+create and create the resource.

Create Virtual Hub

Once the Virtual WAN is created, choose the created resource and click on Hubs under Connectivity and click on New Hub.

Choose the Region. Provide Name, Hub private address space, Virtual hub capacity, Hub routing preference as in the screenshot below. Modify as required.

Provide Gateway scale units, and click on create new under Point to server configuration

Provide a Name for new user configuration and choose tunnel type as OpenVPN in the Basics tab.

Got to Azure Active Directory tab and provide the same value as in the screenshot for Audience. For Issuer https://sts.windows.net/TENANT_ID/ and for AAD Tenant https://login.microsoftonline.com/TENANT_ID/

You can find you Tenant ID by visiting Azure Active Directory home page.

Now click on Review+create and create after validation.

Provide custom DNS as required and click on configure in Address Pools under Configure User Groups to Address Pools Mapping

Provide a Client address pool such as 10.0.0.0/24 and click on Add. Click Review+create and create Virtual Hub.

Setup Azure Firewall

Wait for the Virtual Hub to get created. Once created, choose the Virtual Hub and click on Create under Azure Firewall.

In convert existing hubs, choose the Hub Name if it is not chosen already and choose Azure Firewall tab.

Enable Azure Firewall and choose the tier as Standard. Provide required number of IP addresses in Specify number of Public IP addresses. We are providing 2 in this case. Click on Next.

Skip modifying this section. Now click on Review+confirm to validate and create.

Click confirm to create.

Create Azure Firewall Policy

Visit Firewall Manager console. Click on Azure Firewall Policies.

Choose the Resource group, provide policy name and region. Choose Policy tier as Standard.

Click on Rules tab and add a rule collection. Provide a Name. Choose Rule collection type as Network and set Priority as required. Create a rule by providing Name, source as *, destination as * and choose all protocols in protocols section. Click on Add.

Click on Review+create and create the policy.

Associate Virtual Hub with Firewall Policy

We need to associate the created policy to the Virtual Hub. For that click on Azure Firewall Policies and choose the newly created policy. Click on Manage associations and click Associate hubs.

Select the hub that we created previously and click on Add.

Update Routing Intent & Routing Policies

Visit Virtual WAN console and click on Hubs.

Click on the previously created Hub.

Click on Routing intent and Routing Policies from the left side menu. Choose internet traffic as Azure Firewall and Next Hop Resource as the Virtual Hub that is created previously. Click on Save.

Update Route Tables

Click on Route Tables from the left side menu. Click on the Default Route Table.

VPN configurations are over now. To download the VPN client configurations, follow these steps.

Download VPN Configuration

Visit Virtual WAN console and click on User VPN configurations. Choose the User VPN configurations. Click on Download virtual WAN user VPN profile.

Choose Authentication type and click on Generate and download profile.

Modify VPN Configuration

In Windows, VPN won't work properly with route 0.0.0.0/0. We need to add routes 0.0.0.0/1 and 128.0.0.0/1 in the vpnconfig xml file.

Unzip the downloaded file and navigate to AzureVPN folder.

Open the file using any text editor. Check whether there are routes and DNS added or not. If they are not added, you need to update it manually.

Find and remove remove the line

  <clientconfig i:nil="true" />

Add the following configurations after </clientauth>

<clientconfig>
    <includeroutes>
        <route>
            <destination>0.0.0.0</destination><mask>1</mask>
        </route>
        <route>
             <destination>128.0.0.0</destination><mask>1</mask>
        </route>
    </includeroutes>
</clientconfig>

Save the xml file.

VPN Client Setup - Apple macOS

To connect to the VPN from the client macOs machine, following are the steps.

Find Azure VPN client in App Store

Once installed, open the application.

Click on Import to import VPN xml file that we downloaded. Choose the xml file and click Open.

Click on Save.

Click on Allow if "Azure VPN Client" Would Like to Add VPN Configurations comes. Click on Connect.

Verify the access to VNet and Internet resources.

VPN Client Setup - Microsoft Windows

To connect to the VPN from the client Windows 11 machine, following are the steps.

Find Azure VPN Client from Microsoft Store.

Once installed, open the application. Click on Import to import VPN xml file that we downloaded. Choose the xml file and click Open.

Click on Save.

Click on Connect.

Verify the access to VNet and Internet resources.

In this blog, we will be installing Zabbix Server, Zabbix UI in Amazon Linux 2 arm64 instance. As of now there is no Zabbix package(rpm package) available for arm64 RHEL based OS. The only option is to build zabbix from source.

Launch an Amazon Linux 2 arm64 instance. In this case, I’m launching a t4g.small instance.

Depending on the number of hosts and requests, you may modify the instance type, include RDS instance if necessary, etc.

SSH in to the EC2 instance and issue the following commands to create Zabbix user and group.

sudo groupadd --system zabbix
sudo useradd --system -g zabbix -d /usr/lib/zabbix -s /sbin/nologin -c "Zabbix Monitoring System" zabbix

Create a directory for zabbix source build

sudo mkdir -m u=rwx,g=rwx,o= -p /usr/lib/zabbix

Download latest available version of Zabbix source from Zabbix's official repository

https://cdn.zabbix.com/zabbix/sources/stable/

In this case we are dowloading Zabbix 6.0 LTS. If you want to setup Zabbix 6.0, run below commands. Else modify the URL.

wget https://cdn.zabbix.com/zabbix/sources/stable/6.0/zabbix-6.0.5.tar.gz -O /tmp/zabbix-6.0.5.tar.gz
tar -xvf /tmp/zabbix-6.0.5.tar.gz -C /tmp
sudo mv /tmp/zabbix-6.0.5/* /usr/lib/zabbix

Modify the ownership of the source files copied to /usr/lib/zabbix

sudo chown -R zabbix:zabbix /usr/lib/zabbix

Now we need to install MariaDB Server in the EC2 instance. If you would like to setup an RDS instance instead of this, skip this step.

sudo amazon-linux-extras install mariadb10.5=latest -y

Start and enable MariaDB Server.

sudo systemctl start mariadb
sudo systemctl enable mariadb

Set root password for MariaDB Server by issuing the below command. Provide a secure password and skip all other settings if you are not aware of it.

sudo mysql_secure_installation

Login to MariaDB server.

mysql -uroot -p

Create a database and user for Zabbix Server.

create database zabbix character set utf8mb4 collate utf8mb4_bin;
create user 'zabbix'@'localhost' identified by 'PASSWORD';
grant all privileges on zabbix.* to 'zabbix'@'localhost';

Restore Zabbix Server schema to the database.

mysql -uroot -p zabbix </usr/lib/zabbix/databases/mysql/schema.sql
mysql -uroot -p zabbix </usr/lib/zabbix/databases/mysql/images.sql
mysql -uroot -p zabbix </usr/lib/zabbix/databases/mysql/data.sql

Install the necessary packages to build Zabbix Server.

sudo yum group install "Development Tools" -y
sudo yum install pcre-devel.aarch64 libevent-devel.aarch64 glibc-static.aarch64 zlib-devel.aarch64 OpenIPMI ipmitool libssh2-devel.aarch64 fping libcurl-devel.aarch64 libxml2-devel.aarch64 net-snmp gnutls gcc automake autoconf OpenIPMI-devel mariadb-devel net-snmp-devel 

Install the necessary packages to build Zabbix UI.

sudo amazon-linux-extras install php7.2 -y
sudo yum install php-gd php-bcmath php-ctype php-libXML php-xmlreader php-xmlwriter php-session php-sockets  php-mbstring  php-gettext php-ldap php-openssl php-mysqli php-oci8 php-mbstring go 

Here we are installing Apache as web server for Zabbix UI. Install and enable the service.

sudo yum install httpd
sudo systemctl start httpd && systemctl enable httpd

It's time to build Zabbix from the source. In the following configuration file add DBName, DBUser, DBPassword and save the file.

sudo vim /usr/lib/zabbix/conf/zabbix_server.conf

Build Zabbix by issuing the following commands.

sudo /usr/lib/zabbix/configure --enable-server --enable-agent --with-mysql --enable-ipv6 --with-net-snmp --with-libcurl --with-libxml2 --with-openipmi
sudo make install

After the build is successful, create a systemd service for Zabbix server. Add the following lines in /etc/systemd/system/zabbix-server.service.

[Unit]
Description=Zabbix Server
After=syslog.target network.target mariadb.service
 
[Service]
Type=oneshot
User=zabbix
ExecStart=/usr/local/sbin/zabbix_server
ExecReload=/usr/local/sbin/zabbix_server -R config_cache_reload
RemainAfterExit=yes
PIDFile=/var/run/zabbix/zabbix_server.pid

[Install]
WantedBy=multi-user.target

Reload the newly created systemd file.

sudo systemctl daemon-reload

Start and enable Zabbix Service

sudo systemctl start zabbix-server.service && sudo systemctl enable zabbix-server.service

Now deploy Zabbix UI

sudo mkdir /var/www/html/zabbix
sudo cp -r /usr/lib/zabbix/ui/* /var/www/html/zabbix
sudo chown -R apache:apache /var/www/html/zabbix

Restart php-fpm and apache to complete the deployment

sudo systemctl restart php-fpm
sudo systemctl restart httpd

Zabbix server setup is now complete. You can now access your Zabbix UI from http://IP_ADDRESS/zabbix

If you receive connection timeout error from the browser, please check your instance's security group and confirm whether 80 port is open to your IP address or not.

Why AWS Nitro System ?

Before the invention of the AWS Nitro system, there were several challenges and limitations that users faced in the realm of cloud computing. Major issues are ,

• Traditional virtualization technologies used in cloud computing often suffered from limited hardware utilization.
• The hypervisor layer introduced performance overhead due to the additional processing required for virtualization.
• Traditional virtualization architectures often had shared networking and storage resources, which could become bottlenecks when multiple VMs competed for these resources and this leads to network and storage performance.
• Traditional virtualization approaches creates some security risks, as any vulnerabilities in the hypervisor could potentially compromise all the VMs running on the server.

Many of these issues were resolved with the advent of the AWS Nitro system, which offloaded a number of virtualisation operations to Specialised hardware accelerators. This made it possible for resource allocation in AWS cloud instances to be done more effectively while also enhancing security and speed.

What is AWS Nitro System?

AWS introduced nitro system to learn more about the underlying architecture of an EC2 instances at the virtualisation level. It was introduced in 2017 as a part of continuous innovation of AWS. Initially it was introduced in C5 instance types. The nitro system essentially is the platform that powers the latest and greatest of the next generation EC2 instance types and it effectively operates as a cluster of components. It was launched back in 2017, and all-new instance types since then have been using the AWS Nitro System.

Since Nitro is only used by AWS, it is actually a combination of many parts that uses both custom hardware and software. In addition to being able to satisfy the demands of their customers, it was created to enable AWS to accelerate their rate of innovation.

Components of Nitro System

1. Nitro cards

AWS Nitro Cards, also known as AWS Nitro Acceleration Cards, are specialized hardware components developed by Amazon Web Services (AWS) as part of their Nitro system architecture. These cards are designed to offload specific tasks related to virtualization, networking, and storage, thereby improving the performance, security, and efficiency of AWS EC2 instances. There are 4 different type of nitro cards,

• VPC Networking Nitro card
• EBS Nitro Card
• Instance Storage Nitro card
• Nitro Card controller, or systems controller card

2. Nitro Security Chips

The Nitro Security Chip, the second essential part of the Nitro System, is a unique micro-controller that is physically affixed to the host's motherboard and is used to safeguard hardware resources and enforce the hardware root of trust.The chip itself can only be manipulated and written to by the Nitro card controller, not an instance, and it traps all I/O to non-volatile storage, stores system boot information to enable measurement and validation checks, and traps all I/O to non-volatile storage.  The security chip also needs to make sure that all hardware interfaces are being watched.

3. Nitro Hypervisor

The Nitro Hypervisor which is KVM-based, and is the last key component of the Nitro system. The amount of work the data plane covers and controls, as well as the fact that they take on many of the virtualization responsibilities that a traditional hypervisor would typically handle, are two major advantages of these Nitro cards that I have just described.  AWS has been able to reduce the Nitro hypervisor to its essential elements, leaving only features and components that are absolutely necessary thanks to the transfer of responsibility to the Nitro cards.  As a result, the Nitro hypervisor is never active unless an instance requests it to act on the instance's behalf. As a result, the Hypervisor was created specifically for AWS for the Nitro system, and the work offloading to the Nitro cards has made the Nitro system more efficient.

Benefits of AWS System

The AWS Nitro system, which includes AWS Nitro Cards, provides several benefits for users of Amazon Web Services (AWS) EC2 instances. Some of the key benefits of AWS Nitro System are:

• Enhanced Performance
• Improved Security
• Efficient Resource Utilization
• Enhanced Network Performance
• Accelerated Storage
• Simplified Maintenance and Updates
• Scalability and Agility

Cassandra by default provides a way to take backups of individual keyspaces with the following command.

COPY table_name to 'table_name.csv' WITH HEADER=TRUE

Doing this for for an entire Cassandra cluster with multiple keyspaces (tables) will be difficult. Following set of commands can make this easier for you

First, create a folder to store the backup CSV files

mkdir cassandrabkp

cd into the folder

cd cassandrabkp

Assuming that you are using bash, and you have installed cqlsh and sed, the following command will find the tables and use them one by one to create a backup CSV file for each keyspace.

for i in $(cqlsh 192.168.134.132 -e "DESCRIBE SCHEMA" | grep "TABLE" | sed 's/CREATE TABLE //g' | sed 's/ (//g') ; do echo "cqlsh 192.168.134.132 -e \"COPY $i to '$i.csv' WITH HEADER=TRUE\"";done | bash

The above process will take some time depending on the size.

Now you can cd out of the folder and then compress it to save space. Since it is just texts, from my experience, you can compress a 13GB backup folder to 2.5GB

cd ..

tar -cvzf cassandrabkp.tar.gz cassandrabkp/

You can extract it later with the following command

tar -xvzf cassandrabkp.tar.gz

Previously Cocoapods in mac was installed using gem

sudo gem install cocoapods

This now throws errors when running pod install. To fix this, uninstall the Cocoapods installed using gem with the following command

sudo gem uninstall cocoapods

Now install cocoapods using home brew with the following command

brew install cocoapods

If you see the error 'command not found' when you run the above command, you can install brew with the following command and follow the on screen instructions to enter the password when required

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)"

You should now run the following command to add brew to your PATH

echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> /Users/vignesh/.zprofile

Make sure to change 'vignesh' to your user name

To load the new PATH, run the following command

eval "$(/opt/homebrew/bin/brew shellenv)"

That's it! You should have Cocoapods running on your M1 Mac.

Using a good password is one of the first steps to a better security practice. The most easiest way to ensure that the users use the best password is to enforce it at the system level.

Following is the command that you can use to ensure that your Amazon Linux 2 server uses a standard password policy.

Password policy

Command

The following command will set the above password policy on the server.

sudo authconfig --passminlen=14 --enablefaillock --faillockargs="deny=6 unlock_time=1800" --passminclass=4 --update

Verification

You can verify that the above password policy is implemented by checking the system-auth configuration file. Please note that changing this file directly will not be persistent since the changes will be reverted when the authconfig command is run again with a different configuration.

cat /etc/pam.d/system-auth

#%PAM-1.0
# This file is auto-generated.
# User changes will be destroyed the next time authconfig is run.
auth        required      pam_env.so
auth        required      pam_faildelay.so delay=2000000
auth        required      pam_faillock.so preauth silent deny=6 unlock_time=1800
auth        sufficient    pam_unix.so nullok try_first_pass
auth        requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth        required      pam_faillock.so authfail deny=6 unlock_time=1800
auth        required      pam_deny.so

account     required      pam_faillock.so
account     required      pam_unix.so
account     sufficient    pam_localuser.so
account     sufficient    pam_succeed_if.so uid < 1000 quiet
account     required      pam_permit.so

password    requisite     pam_pwquality.so try_first_pass local_users_only retry=3 authtok_type=
password    sufficient    pam_unix.so sha512 shadow nullok try_first_pass use_authtok
password    required      pam_deny.so

session     optional      pam_keyinit.so revoke
session     required      pam_limits.so
-session     optional      pam_systemd.so
session     [success=1 default=ignore] pam_succeed_if.so service in crond quiet use_uid
session     required      pam_unix.so

References

How to Enforce Password Policies in Linux (Ubuntu / CentOS)

In this guide we will learn how to enforce password policies in Linux based operating systems like CentOS, RHEL, Ubuntu and Debian.

LinuxTechi |James Kiarie

authconfig(8) - Linux man page

authconfig provides a simple method of configuring /etc/sysconfig/network to handle NIS, as well as /etc/passwd and /etc/shadow, the files used for shadow password support.

Linux man page

Lock User Account After n Failed Login attempts in Linux

Learn how to lock user accounts automatically after n incorrect or failed login attempts in Linux distributions like CentOS, RHEL, Fedora, Debian and Ubuntu.

LinuxTechi |Pradeep Kumar

How to enable faillock using authconfig - Red Hat Customer Portal

Executing authconfig command removes the faillock entries from PAM files. Configure faillock for persistent settings in PAM files.

Red Hat Customer Portal

CentOS 7 : Set Password Rules : Server World

How to set password complexity in redhat 7.5?

I am trying to set a password policy complexity on red-hat 7.5.i want that every user that will try to change his password will have to use password with at least - (1 Lower , 1 Upper , 1 Digit , 1

Server FaultChinChen

Sometimes, the data in your servers are so valuable that having ssh key with passphrase, IP restrictions and sudo password all together won't just cut it for your organisation's security policy.

In this case, you'll have to set up MFA for your server, and following are the steps.

Here I am setting a Google Authenticator based MFA for Amazon Linux 2 in addition to a password based authentication for the user. You can use key based authentication also with small changes in configuration.

Installation of Google Authenticator

First, enable the epel repo in AL2

sudo amazon-linux-extras install epel -y

Now you can install the google-authenticator package

sudo yum install google-authenticator -y

Creating MFA

Now run the google-authenticator command and give the following responses

[ec2-user@vm_al2 ~]$ google-authenticator 

Do you want authentication tokens to be time-based (y/n) y
Warning: pasting the following URL into your browser exposes the OTP secret to Google:
  https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/ec2-user@vm_al2%3Fsecret%3DKPPS2C44OQMUYKQCNEG45SVIL4%26issuer%3Dvm_al2

Your new secret key is: KPPS2C44OQMUYKQCNEG45SVIL4
Your verification code is 061304
Your emergency scratch codes are:
  53063172
  99973171
  81062482
  92606712
  26718859

Do you want me to update your "/home/ec2-user/.google_authenticator" file? (y/n) y

Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y

By default, a new token is generated every 30 seconds by the mobile app.
In order to compensate for possible time-skew between the client and the server,
we allow an extra token before and after the current time. This allows for a
time skew of up to 30 seconds between authentication server and client. If you
experience problems with poor time synchronization, you can increase the window
from its default size of 3 permitted codes (one previous code, the current
code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes
between client and server.
Do you want to do so? (y/n) n

If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting? (y/n) y

Configuring the server to use MFA

Now configure the PAM module to use Google Authenticator while logging in

sudo vim /etc/pam.d/sshd

Edit the file to include the following line

auth required pam_google_authenticator.so nullok

The nullok allows the users without MFA configured to login without MFA. Only the users with MFA will be asked for an MFA. This can be removed once all the users in the server has set the MFA.

Now edit the sshd configuration to as for the OTP challenge

sudo vim /etc/ssh/sshd_config

Edit the configuration to change the ChallengeResponseAuthentication to yes

ChallengeResponseAuthentication yes

Since I'm configuring this for password authentication only and not ssh key authentication, my sshd_config file has PasswordAuthentication yes instead of the default PasswordAuthentication no

Now you can restart the sshd process

sudo systemctl restart sshd

Conclusion

That's it, you can login with the MFA to your server

vignesh@workstation:~$ ssh ec2-user@192.168.122.125
Password: 
Verification code: 
Last login: Fri Apr 29 04:59:47 2022 from 192.168.122.1

       __|  __|_  )
       _|  (     /   Amazon Linux 2 AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-2/
55 package(s) needed for security, out of 95 available
Run "sudo yum update" to apply all updates.
[ec2-user@vm_al2 ~]$

You can also copy the .google_authenticator file from your user's home folder to your local machine to keep the key, emergency use OTPs etc. safe.

[ec2-user@vm_al2 ~]$ cat .google_authenticator 
KPPS2C44OQMUYKQCNEG45SVIL4
" RATE_LIMIT 3 30 1651209615
" DISALLOW_REUSE 55040320
" TOTP_AUTH
53063172
99973171
81062482
92606712
26718859

Also if you want to use only key based authentication,

Open the pam file

sudo vim /etc/pam.d/sshd

and comment the following line

#auth substack password-auth

References

Setting up multi-factor authentication on Linux systems

Pluggable Authentication Modules allow Linux to work with Google Authenticator and other OTP tools to add two-factor security to your system.

Enable SysadminKeerthi Chinthaguntla

OpenSSH server: how to configure keyboard-interactive authentication

How can I configure the OpenSSH server (on Ubuntu) to allow keyboard-interactive but not password authentication? I know that public-key-authentication is the preferred one, but I want to test a u...

Super UserMike L.

How to Secure Your Instances with Multi-factor Authentication | Amazon Web Services

An AWS Solutions Architect walks through implementing an additional layer of authentication security for your EC2 instances by requiring two-factor authentication for administrators to use SSH to connect.

Amazon Web ServicesRoshan Kothari

Enabling Two Factor Authentication for EC2 SSH - AWS MFA Setup

AWS MFA Setup -How to Secure your SSH Authentication to EC2 instance with Google Authenticator multi factor authentication(MFA). aws ec2 ssh mfa. AWS Google auth. AWS SSH MFA Configuration. SSH Security with AWS EC2. Configure Google Authenticator in Ec2 instance and how to securely SSH into your EC…

Middleware InventorySarav AK

The first problem most people face after their custom Wordpress installation is the file upload limit.

There are various ways you can use to fix it. Following is what I use and which seems to work always. I use the Wordpress docker container, so your mileage may vary if you are using manual installation.

The htaccess method

For me, putting the following entry into the .htaccess file in the wordpress installation folder seems to fix the issue.

php_value upload_max_filesize 1000M
php_value post_max_size 2000M
php_value max_execution_time 300
php_value max_input_time 300
php_value memory_limit 3000M
php_value file_uploads On

I used the WP file manager and edited this file. This is fine since I had deployed it in AWS ECS Fargate with EFS storage backend and hence the file change will persist.

If you are only putting only selected folders to persistent storage instead of the entire document root, then you'll have to make this change at the image level.

Recently I had a requirement where I had to deploy multiple angular/react/vue.js apps and an API to separate paths and serve it through Nginx.

Following is the configuration that I used.

server{
        server_name nginxangular.vigneshn.in;

        location /api{
                proxy_pass          http://127.0.0.1:8080;
                proxy_http_version  1.1;
                proxy_set_header    Upgrade             $http_upgrade;
                proxy_set_header    Host                $host;
                proxy_set_header    X-Real-IP           $remote_addr;
                proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
	}
    location /admin {
                alias /data/app/ui/admin;
                index index.html index.htm;
                try_files $uri $uri/ /admin/index.html;
                }
    location /client {
                alias /data/app/ui/client;
                index index.html index.htm;
                try_files $uri $uri/ /client/index.html;
                }
    location /business {
                alias /data/app/ui/business;
                index index.html index.htm;
                try_files $uri $uri/ /business/index.html;
                }
    location / {
                alias /data/app/ui/common/;
                index index.html index.htm;
                try_files $uri $uri/ /index.html =404;
                }
}

The admin,client,business,common are the angular deployment folders. The common folder is served from / and the rest of them are served from their appropriate paths.

Ever wondered how you can watch all the queries running in the database by someone? Here's how.

Let's Begin

Visit RDS console and choose the RDS instance. If the RDS instance is attached to a default parameter group, you have to create a new custom parameter group from the default parameter group.

After the custom parameter group is created, attach it to the RDS instance. Expect a downtime while you replace the existing parameter group. There won't be any downtime if you modify a parameter in the existing custom parameter group.

Modifying Parameter Group

1. Beginning with general_log. The default value will be disabled(0) or blank. Modify it to 1

2. Next parameter, log_output asks you where to save these logs, in this case I'm choosing file. The logs can be watched in real time or downloaded from the RDS console.

Click on Save Changes and wait for some time for the RDS to modify the parameter group.

Watching Logs

Choose Logs & Events tab. Scroll down and choose the general query log from Logs section and click on either Watch or Download.

When I ran a simple query in MySQL client, RDS logs it as in the given image.

One of the easiest metrics to check if your application is running is to see if it is loading properly and that all its features are working fine. There are many monitoring tools like Zabbix, Nagios, DataDog etc that allows us to do this. To add on to that and with more customisability with the usage of Lambda, the AWS cloudwatch Sythetics Canaries is an exceptional monitoring service to monitor custom APIs or websites.

AWS Synthetics Canaries actually use Lambda in the background with selenium or puppeteer scripts to do the monitoring. Previously, we had to set up a cloudwatch event, set up Lambda, write custom code for the Lambda and then the Lambda had to push the metrics or logs to cloudwatch. Synthetics Canaries simplifies this entire process and makes the configuration of API or web monitoring easy with just a coouple of clicks.

Why I used this?

I had to use this service because I had a requirement from my customer to monitor the TAT (Turn Around Time) of an external API for an application running in AWS in a VPC with external calls going through a Managed NAT. From that, I had 3 ways to do it.

• Zabbix web check
• Zabbix user parameter
• AWS Cloudwatch Synthetics Canaries

I tested each of the methods and provided the pros and cons with each method and following were my findings.

Zabbix web check

Zabbix has an option to do web check wherein we can create steps and configure the method, URL, data, headers and other parameters of an HTTP request and use that to monitor the retured response, status codes, latency etc.

One issue I found with this method is that in Zabbix, the webcheck is done from the Zabbix server itself (or proxy server if the host having the webcheck is monitored by a proxy), this means that the people running the external API has to allowlist the Zabbix server IP also which is an additional step. We also skip the route via the Managed NAT which is being used by the rest of the application servers.

So this methods was not recommended

Zabbix user parameter

In this method, a custom user parameter is added to any one of the application server. User parameter is a specific custom key that can be monitored by Zabbix. This is configured in the Zabbix agent of the host which we will add the monitoring item to.

The user parameter is a custom script or command that gives the required output which can then be used directly or parsed by Zabbix.

In this case, we will be using the following curl command to get the total time or latency of the API request as a float value.

curl -w "%{time_total}\n" -o /dev/null -s "https://vigneshn.in/"

One advantage with this method is that the request will go through the NAT gateway since the command is run from the application server running in the VPC

The problem with this method is that the configuration for this exists in the Zabbix server and a specific application server in which that user parameter is configured. This is a problem if that application server goes down or if there is an infrastrucutre or application change that makes that server obsolete. This also will be a problem considering that running that command will use some CPU, memory and network of the application server.

So this method was also not recommended.

AWS Cloudwatch Synthetics Canaries

Since both the above solutions had their own specific technical limitations, AWS Cloudwatch Synthetics Canaries was tested.

In this case, it was configured through the AWS console by entering the required API URL, data, headers etc. and it was complete. I was able to configured the VPC, Subnet, Security group etc which the backend Lambda should use and this means that the request will always go through the NAT gateway which shows the latency via NAT and also no additional IP has to be allowed in the external API.

Since this does not depend on any of the internal servers, any downtime in them will not affect the monitoring. Since two subnets were selected in different availability zones, even if an availability zone goes down, the monitoring will still work.

The only problem with this is the additonal cost associated with this, but it is negligible.

How to set up the Synthetics Canaries

First open the cloudwatch dashboard and on the left bottom, there is an option named Synthetics Canaries. Clicking that will open up the Canaries page with the status dashboard. Here, click on Create canary button

When you click the Create canary button, you'll be presented with the options on how to create it. You can use a blueprint (easy method), you can use an inline editor (customisable with your own scripts) or you can load the scripts from S3.

There are some typically used monitoring methods in the "Use a blueprint" option like Heartbeat monitoring, API canary, Broken link checker etc.

In our case, we will be using the API canary option since we are planning to check the latency of an API call. I have also put a name for the canary as "canary-demo".

Now it is time to set the configuration for the API call. As an example, I am just sending a GET request to https://www.google.com with two required headers (Otherwise we will get  bad request as the response).

Method: GET
URL: https://www.google.com
Headers:
	- user-agent : curl/7.74.0
    	- accept : */*

You also have an option to put request data but we are skipping it in this example.

If you are putting sensitive header like Authorization, the console will highlight it in red and show a button below to redact that information from coming in the logs, HAR etc. You can press that button or create the canary as is depending on your requirements.

Once you have set up the HTTP request, you can choose the Runtime version. I am using sys-nodejs-puppeteer-3.2 but it is only signifincant if you are modifying the script before running. In that case, you can choose the required language.

Now it is time to choose the schedule to tell the Sythetics Canary how frequently or at what time it should run. You can set it to run continously one a specific date or at regular intervals of time, you can put in a CRON expression, or you can run it only once.

In my case, I am choosing the "Run continously" option because I'm lazy to find the cron expression. I always forget the format.

I am also checking the option to start it immediately after creation.

There are some additional configuration also to set a timeout but I am leaving it as is.

There is also an option to specify how long the failure data and the success data should be retained. I am putting it as 1 month for both success and failure data.

The data that is collected during each canary run is stored in an S3 bucket. By default, an S3 bucket will be created with the format given in the screenshot

s3://cw-syn-results-AWSAccountNumber-Region/canary/...

You do have the option to change it but that will require you to already have set up an IAM role with the correct permission to write to that S3 bucket.

You can either configure the canary to create a new role or select and existing role in the next "Access permissions" step. Do note that once the S3 bucket is edited, then the "Create a new role" option will not be selectable.

You can also create CloudWatch alarms which I am skipping now but it is required if you need to get alerts when the API latency crosses a set threshold.

As I said perviously, my requirement had to have the lambda call the external API from within the VPC since the application servers were inside the VPC and they were sending the request through the NAT.

In this case I am not configuring the VPC for this demo because I configured google.com as the API endpoint and since it is in the public internet, I will need a NAT in the VPC since lambda can talk to public internet from inside the VPC only if there is a NAT gateway attached and as AWS managed NAT gateways are friggin expensive.

When you are giving a custom S3 location, you have to create a service role for Lambda with the following configurations

Path

/service-role/

IAM Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::{bucketname}/{path}/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::{bucketname}"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:CreateLogGroup"
            ],
            "Resource": [
                "arn:aws:logs:ap-northeast-1:038218625917:log-group:/aws/lambda/cwsyn-{canaryname}-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "xray:PutTraceSegments"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Resource": "*",
            "Action": "cloudwatch:PutMetricData",
            "Condition": {
                "StringEquals": {
                    "cloudwatch:namespace": "CloudWatchSynthetics"
                }
            }
        }
    ]
}

Trust relationship

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

If you are providing VPC configurations to make requests from inside your VPC, remember to also attach the AWSLambdaVPCAccessExecutionRole to the created IAM Role. This is required by the Lambda running in the backend of the Synthetics Canaries to create network interfaces inside the VPC with the given configuration.

Now you can tag the canary and if you want, you can also enable Active tracing with AWS X-Ray. I am keeping mine checked.

That is it, you can click the Create canary button as you see in the above screenshot and your canary will be created. You will be able to see the failure and success status in the canary dashboard.

If you make any mistakes, you have the option to edit the configuration after the entre thing is created.

Following are the steps that I took to uninstall bloatware from my Samsung M30s phone without root.

Requirements

1. Phone (I use Samsung Galaxy M30s)
2. PC (I used Linux)
3. Data cable (I use Type C)

Steps

Connect the phone to the PC

Connect the phone to the PC with the data cable and allow or deny the storage access based on your need since we won't be using that for this purpose.

Enable USB Debugging

Once it is connected, go to your settings and take the developers options and enable the USB Debugging option.

If you don't see the developer options, you will have to enabled that by going to Settings -> About phone -> Software information and tapping the Build number 5 times.

When you enable USB debugging, it might show a confirmation saying that it is for development purpose and all, you can just tap OK.

Install ADB

Use your favourite package manager to install the adb package. I use PopOS which is an Ubuntu based (Debian) OS and it has apt.

sudo apt install adb

Android Debug Bridge is a CLI tool that we use to communicate with the device. It provides a shell interface which we will use in the following steps to run commands.

Authorise the PC

Once ADB is installed run the following command to check if your device is listed

adb devices

Running this command will give the following output

List of devices attached
W5TP428P3MS	unauthorised

You will now be able to see a prompt on your phone asking you if you want to authorise the PC as an USB debugging device. Confirm that and when you run the command again, you should see the following output.

List of devices attached
W5TP428P3MS	device

List the packages

Now we can start running the commands on the ADB shell and it will run inside the device.

For opening the shell, assuming you only have a single device connected, you can run the following command

adb shell

This will give you the following prompt

vignesh@vignesh-asus-pop-os:~$ adb shell 
m30s:/ $

Now you can list the installed packages by running the following command

pm list packages

This will list he packages as follows

vignesh@vignesh-asus-pop-os:~$ adb shell 
m30s:/ $ pm list packages
package:com.samsung.android.provider.filterprovider
package:com.google.android.apps.subscriptions.red
package:com.sec.android.app.DataCreate
package:com.skype.raider
...
...
...

You can use the grep command to find the package of your interest quickly

pm list packages | grep skype

I wanted to remove the Samsung Pay Mini application and hence my grep keyword was "pay", but here I am using Skype as an example

This showed me the exact package name of that application

m30s:/ $ pm list packages | grep skype
package:com.skype.raider

Uninstall the bloatware

Now that you got the package name, you can uninstall the package with the following command

pm uninstall -k com.skype.raider

If this was successful, it will show success, else it will show failed.

Usually it fails because the app you are trying to uninstall would be a system app. So you can only uninstall it for your own user since we don't have the root access.

This is good because if you uninstall an essentail app like an API or OS package, you can easily fix your phone by resetting it.

To uninstall a system app, you have to specify the user option as follows

pm uninstall --user 0 -k com.skype.raider

Conclusion

You should be able to uninstall all the bloatware with this method without root. Just make sure you don't delete any necessary packages. I once made that mistake and had to reset my phone.

Disclaimer: I am not responsible if you damage your phone doing the above steps. It worked for me and hence I have put this blog post for educational purpose.

I tried to set up JMX monitoring for a couple of Tomcat servers and it took a lot of time than it needed to. So this blog simplifies that process.

The environment I set it up was a trusted network (VPC in AWS) so I did not bother with the security part since everything is restricted using security groups.

Some explanation

I shall try and explain some concepts more clearer than the documentation and relating to my environment

Infra diagram

Zabbix Java Gateway

The Zabbix Java gateway is basically like a Zabbix proxy server but unlike a normal Zabbix proxy server which cannot be nested i.e you cannot set one proxy server behind another proxy server, you can set a Zabbix Java gateway behind a proxy or directly connecting to the Zabbix server.

There is also another difference compared to a Zabbix proxy. You can add multiple Zabbix proxies with the Zabbix server UI, but there can be only one Zabbix Java gateway and it has to be configured in /etc/zabbix/zabbix_server.conf (or /usr/local/etc/zabbix_server.conf) or wherever your Zabbix server configuration is.

Setting up JMX monitoring

Zabbix Java gateway setup

The following was my configuration after I installed the Zabbix Java gateway in the Zabbix server.

Make sure that you start the Zabbix Java gateway service.

JavaGateway=127.0.0.1
JavaGatewayPort=10052
StartJavaPollers=5

Restart the Zabbix server service after updating with the above configuration

Make sure that any internal firewall or security group has the access enabled for 12345 port to the Tomcat server

Tomcat setup

To make tomcat start listening on 12345 port for the jmx connections, add the following options to it.

java.rmi.server.hostname=xxx.xxx.xxx.xxx
com.sun.management.jmxremote.rmi.port=12345
com.sun.management.jmxremote
com.sun.management.jmxremote.port=12345
com.sun.management.jmxremote.authenticate=false
com.sun.management.jmxremote.ssl=false
com.sun.management.jmxremote.registry.ssl=false

So in this case, the Tomcat 1 (192.168.5.1) server will have the following entry in /etc/systemd/system/tomcat.service

[Unit]
Description=Apache Tomcat Web Application Container
After=syslog.target network.target

[Service]
Type=forking

Environment=UMASK=022
Environment=JAVA_HOME=/usr/lib/jvm/jre
Environment=CATALINA_PID=/usr/share/tomcat8/temp/tomcat.pid
Environment=CATALINA_HOME=/usr/share/tomcat8
Environment=CATALINA_BASE=/usr/share/tomcat8
Environment='CATALINA_OPTS=-Xms512M -Xmx3072M -Xss2M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djava.rmi.server.hostname=192.168.5.1 -Dcom.sun.management.jmxremote.rmi.port=12345 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.registry.ssl=false'

ExecStart=/usr/share/tomcat8/bin/startup.sh
ExecStop=/bin/kill -15 $MAINPID

User=root
Group=tomcat

[Install]
WantedBy=multi-user.target

and Tomcat 2 (192.168.6.1) will have the following entry in the tomcat service file

[Unit]
Description=Apache Tomcat Web Application Container
After=syslog.target network.target

[Service]
Type=forking

Environment=UMASK=022
Environment=JAVA_HOME=/usr/lib/jvm/jre
Environment=CATALINA_PID=/usr/share/tomcat8/temp/tomcat.pid
Environment=CATALINA_HOME=/usr/share/tomcat8
Environment=CATALINA_BASE=/usr/share/tomcat8
Environment='CATALINA_OPTS=-Xms512M -Xmx3072M -Xss2M -server -XX:+UseParallelGC'
Environment='JAVA_OPTS=-Djava.awt.headless=true -Djava.security.egd=file:/dev/./urandom -Djava.rmi.server.hostname=192.168.6.1 -Dcom.sun.management.jmxremote.rmi.port=12345 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12345 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.registry.ssl=false'

ExecStart=/usr/share/tomcat8/bin/startup.sh
ExecStop=/bin/kill -15 $MAINPID

User=root
Group=tomcat

[Install]
WantedBy=multi-user.target

As you can see, the java.rmi.server.hostname option is configured with the Tomcat server's own IP address to which the Zabbix server connects to.

I have put ssl and other authentication and security options to false because this is inside a VPC and we have other layers of security in place.

After setting the above configuration in the service file, make sure to restart tomcat with the following commands

sudo systemctl daemon-reload
sudo systemctl restart tomcat.service

Zabbix Server UI config

This section tell you what you have to do in the Zabbix server UI side to configure the Tomcat servers as hosts for monitoring

In this case, it is just like configuring any other Zabbix agent host, but you have to enter the IP 192.168.5.1 (for Tomcat-1) and the port 12345 in the JMX interfaces section

If your configuration were correct, you should see the green JMX indicator

Some problems I faced

SSL error

SSL peer shut down incorrectly: service:jmx:rmi:///jndi/rmi://192.168.5.1:12345/jmxrmi

I got the above error and I fixed it with the help from this answer. This is there in the systemd service file example I have put above.

This showed up as a red JMX indicator with the above error.

Wrong understanding

I thought this was like Zabbix agent itself where we install an agent on all the servers that we have to monitor, but it is not like that. We install Zabbix Java gateway in a single place that can access the JMX port on all the servers we want to monitor.

So we can install the gateway on the Zabbix server itself and when we use 12345 as the JMX port, we can open the firewall from Zabbix agent to the Tomcat server for 12345 port.

Since the Zabbix Java gateway agent listening on port 10052 is inside the Zabbix server itself, it doesn't need to be enabled in the firewall for external access.

I once had a requirement to log bash commands and I checked many solutions like Snoopy, but none of them were proper for a production environment. So the below is the most simple method to add logging to bash in Linux.

Steps

Modify bash config

First step is to modify the configuration file for bash

Open /etc/bashrc with the following command

sudo vim /etc/bashrc

After that, add the following line to the end of the file

PROMPT_COMMAND='history -a >(tee -a ~/.bash_history | logger -p local6.info -t "$USER[$$] $SSH_CONNECTION")'

The above entry basically logs all the bash history (commands) to the local6.info log severity.

This will start working only after logging out and logging in again (Or you can manually source the new bashrc)

Modify Rsyslog config

Now to actually log the bash commands, the Rsyslog configuration must be edited to send all the local6.* logs to the /var/log/secure file

So open up Rsyslog config with the following command

sudo vim /etc/rsyslog.conf

And then change the existing secure log entry

from

authpriv.*                                 /var/log/secure

to

authpriv.*,local6.*                        /var/log/secure

After this, restart Rsyslog with the following command

sudo systemctl restart rsyslogd.service

Conclusion

This will enable bash logging but do note that this will enable only bash logging. If there is any other shell like sh or zsh, this method will not log the commands run using those shells.

Also anyone can put any commands in a script and run it and this will only log the name of the script and not the commands inside the script.