Category Archives: Tech Stuff

Getting real IP while using nginx working behind other nginx

nginx logoFor some reasons you might encounter a situation when you have one nginx working as reverse proxy behind other nginx server. So users send a request to nginx 1, and it sends it to nginx 2. The remote IP of request source as nginx 2 gets is is the IP of nginx 1, and not IP of the user. Here’s how you can fix it. On frontend server (nginx 1) you have to set adding headers:

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

Make sure nginx you have on backend server was built with http_realip_module. You can check it this way:

nginx -V

And add to you nginx 2 configuration these lines:

set_real_ip_from <here goes frontend server IP>;
real_ip_header X-Forwarded-For;

Now nginx on backend will replace all remote IP from frontend server with the one from X-Forwarded-For HTTP header. If you need more information about ngx_http_realip_module you can get it on nginx oficial site.

Email infrastructure made right. Part 3: feedback loops.

In the previous posts (part 1, part 2) we covered some required items which should be implemented to send emails safely. Here’s one more item you should consider before starting sending emails: feedback loops. It works next way. You register on FBL and if some of your recipients mark your mail as spam you will get a special message about it. So you can take necessary actions. You can get a list of available feedback loops on corresponding Wikipedia page. It’s free of charge. For MSN, Zoho and Yahoo you will need their accounts, Google doesn’t have FBL.

Launching Python applications with Gunicorn and systemd

At ServerBeep.com we love Python and use it widely. There are a lot of ways to deploy Python application. One of them is using Gunicorn.

Gunicorn is a Python WSGI HTTP Server for UNIX. So if you have WSGI complaint application you can run it with Gunicorn. It can be, for instance, Django or Flask powered applications.

From the other hand, last versions of Fedora Linux ship with systemd. Systemd is a featured replacement of old SysV system. Among other features there is one which allows to launch the processes after server boot. To use systemd for the applications called app launch create file /lib/systemd/system/gunicorn-app.service:

[Unit]
Description=gunicorn-app

[Service]
ExecStart=/usr/bin/gunicorn -D -n gunicorn-app -w5 tracwsgi:application -b 127.0.0.1:8000 --access-logfile /home/app/log/app.serverbeep.net-access.log --error-logfile /home/app/log/app.serverbeep.net-error.log
Type=forking
User=app
Group=app
Restart=always
StandardOutput=syslog
StandardError=syslog
WorkingDirectory = /home/app/

[Install]
WantedBy=multi-user.target

And enable it:

systemctl enable gunicorn-app.service

Now after server boots you will get your application run. After you start it Gunicorn will handle the requests which are sent to 127.0.0.1:8000. Next step could be nginx or Apache setup as reverse proxy. It means that webserver listens well-known port 80 and serves some files (usually static ones) directly and send rest of requests to Gunicorn. It responses back and webserver provides a reply to a user.

With systemd you don’t have to setup supervisord or other processes monitoring and management tools. Systemd will take care of it. It’s easy and neat way to launch the daemons.

Email infrastructure made right. Part 2: handling bounces.

In our previous post on email infrastructure we were talking about minimum requirements which should be implemented. Today we will cover bounces handling. What do the bounces mean? In short words, it’s a non delivery report. You could get if you send a mail to non-existing user (error code 5.1.1 ) or a user which has full mailbox (5.2.2).

There are kinds of the bounces:

  • soft bounces;
  • hard bounces.

If you get soft bounces it means that you can try to send an email later. And hard bounces mean that email account you are trying to send to is invalid. Such email accounts should not be used for further mail sending. And if you have a lot of subscribers you need a way to get rid of invalid email accounts. But how can you do it? Fortunately there’s a simple way. Probably you’ve heard about VERP. It stands for Variable envelope return path. Here’s how it works (we will cover Sendmail and Dovecot based setup on Centos).

Let’s assume we want to send a mail to joe@example.com from no-reply@serverbeep.com. If there’s no user joe@example.com we will get a bounce message to no-reply@serverbeep.com. But why don’t we collect all bounces in one mailbox? For simplicity of handling bounces we can setup some mail account. Let’s called it bounces@serverbeep.com and setup our LDA to deliver all mails to some user account (it could be system user bounces or some other account depending on your setup). Set up Sendmail to forward all incoming mail sent to bounces@serverbeep.com to system account bounces. Open /etc/aliases files and add next line:

bounces@serverbeep.com: bounces

Tell Sendmail to use Dovecot as LDA:

FEATURE(`local_procmail', `/usr/libexec/dovecot/deliver',`/usr/libexec/dovecot/deliver -d $u')
MODIFY_MAILER_FLAGS(`LOCAL', `-f')
MAILER(procmail)dnl

It should be placed to /etc/mail/sendmail.mc.

Apply Sendmail changes:

cd /etc/mail ; make all restart

If you don’t have Dovecot installed it’s time to install it:

yum install dovecot

Here’s an example of Dovecot configuration file working as LDA:

# 2.0.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.1.1.el6.x86_64 x86_64 CentOS release 6.3 (Final)
disable_plaintext_auth = no
lda_mailbox_autocreate = yes
listen = 127.0.0.1
mail_location = maildir:~/Maildir
mbox_write_locks = fcntl
passdb {
driver = pam
}
postmaster_address = postmaster@serverbeep.com
ssl = no
userdb {
driver = passwd
}
protocol lda {
auth_socket_path = /var/run/dovecot/auth-master
info_log_path = /var/log/dovecot/delivery-info.log
log_path = /var/log/dovecot/delivery.log
postmaster_address = postmaster@serverbeep.com
}

(Make sure you have /var/log/dovecot and dovecot has the permissions to write there).

Well, since we prepared everything let’s get back to VERP. To get a bounces while sending a mail to joe@example.com to bounces user maildir we need to add Return-Path header to our mail. Short example in PHP:

<?php
mail("joe@example.com", "Message subject", "Message body", "From: no-reply@serverbeep.com\n", "-fbounces+joe=example.com@serverbeep.com");
?>

So, there will be additional header in the email:

Return-Path: bounces+joe=example.com@serverbeep.com

In this case if there’s no user joe@example.com we will get a bounce message in our POP3/IMAP4 account bounces from where the messages could be easily fetched with simple Python script.

import imaplib
M = imaplib.IMAP4('127.0.0.1')
M.login('bounces', 'strongest_password')
M.select()
typ, data = M.search(None, 'ALL')
for num in data[0].split():
typ, data = M.fetch(num, '(RFC822)')
print 'Message %s\n%s\n' % (num, data[0][1])
M.close()
M.logout()

Once you get the messages you can parse it to get a list of invalid email accounts and make a modification of your users table in the database.

Using svn+ssh with non-standard port

If you use svn over SSH you may encounter an issue related to non-standard SSH port:

[dev@serverbeep web]$ svn checkout svn+ssh://dev@dev.devserver.com/home/svn-dev/
svn: To better debug SSH connection problems, remove the -q option from 'ssh' in the [tunnels] section of your Subversion configuration file.
svn: Network connection closed unexpectedly
[dev@serverbeep web]$

It’s because ssh client tries to connect to default SSH port 22. The solution is quite easy. You just need to add to your svn configuration file ~/.subversion/config next line:

ssh = $SVN_SSH ssh -q -p 2201

It should be in section [tunnels]. And this is it. Now you can checkout or update or whaterver you need to do with your remote SVN repository.

Email infrastructure made right. Part 1: basic requirements.

If you send emails quite often like we do at ServerBeep.com (we send email notifications if something bad happen with our customers’ websites or services), you had better to prepare your email setup before you start. It’s better to do it before than to solve the issues after. There are few items described below which should be consider as required before you start.

Check you IP addresses history

Sender Score and Sender Base allow you to check your IP history. If you find any issues you’d better consider changing it.

MX and PTR DNS records

To send a mail you have to setup PTR and MX DNS records properly so your IP address should point to your domain. A lot of SMTP server would reject your mail if you don’t have correct PTR record.

[root@sb1 ~]# host 88.198.20.28
28.20.198.88.in-addr.arpa domain name pointer serverbeep.com.
[root@sb1 ~]#

SPF

SPF stands for Sender Policy Framework. It allows you to tell other SMTP servers which IP and/or domain are allowed to send a mail from your domain. And it’s quite easy to setup. There’re a lot of online services which could help you with it.

Here’s for instance, our SPF record:

serverbeep.com. 42008 IN SPF "v=spf1 ip4:88.198.20.28 ip6:2a01:4f8:130:32a4::28 ~all"

Besides, it would be better to have the same one in TXT record. Just in case.

If you check the headers from the mail you received to your Gmail account you can see how Google treats SPF:

Received-SPF: pass (google.com: domain of no-reply@serverbeep.com designates 2a01:4f8:130:32a4::28 as permitted sender) client-ip=2a01:4f8:130:32a4::28;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of no-reply@serverbeep.com designates 2a01:4f8:130:32a4::28 as permitted sender) smtp.mail=no-reply@serverbeep.com; dkim=pass header.i=@serverbeep.com

Using Gmail is the simplest way to check if you set up SPF properly. Besides there are a lot of other ways to check online. Just google for ‘spf check’.

DomainKeys Identified Mail

To setup DKIM you need to generate public and private keys. Public should be added to your domain zone while private key should be securely stored on you server. SMTP server, Sendmail for instance, uses private key to add a digital signature which can be used for email validation.

Here’s an example of DKIM record:

default._domainkey.serverbeep.com. 41765 IN TXT "v=DKIM1\; k=rsa\; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDXFOEq2OCK1Bpp7YpId3nSgJX8fcfl5bx3a9aiDHm5nCAx0ZmAPacOoo+dmFlBfbcAIbi2BPons6w/uIW4rSeFAuybGw04/wMOkVit1OJPoiCsGW9BJLPeMnez+6m32zv3drVjgeywxtVDNbQphOIJdk4S88O2hlWOsusYv4sEdQIDAQAB"

The same as for SPF, you could check your DKIM by sending a mail to your Gmail account. Look for DKIM-Signature and Authentication-Results headers.

Necessary mail aliases: abuse@, postmaster@ and fbl@

If there are any issues other postmasters would send a mail to some of these aliases. So again, you should definitely have them set up.

Smart Network Data Services by Microsoft

Register on SNDS to be informed if there’re any issues with Hotmail.

Of course, it’s not all required steps. But probably the most important ones.

Debugging HTTP requests with Flask and netcat

ServerBeep.com is in ongoing development process. To meet your requirements we try to do our best to provide the best and the simplest website and monitoring service. This short post is dedicated to debbuging HTTP requests with Flask and netcat.

1. Flask

Ok, so we need to see what the webserver gets. With Flask it’s pretty simple. Flask is light but power Python framework built on top of Werkzeug and has a lot of extensions and features. It’s not so featured framework as Django but for simple applications it fits very well. (Before we were adepts of web.py and were quite happy with it. Now some of our internal applications are still powered with it, but Flask seems to have more active community and is under more dynamic development at the moment). This a complete code of how to get a request as Flask it sees:

from flask import Flask, request
app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])

def hello_world():
    from pprint import pprint
    pprint (request.form)
    # If you need headers just comment out next line. 
    # pprint (request.headers)
    return 'UP'

if __name__ == '__main__':
    app.debug = True
    app.run()

And this is it. By default Flask application waits for the requests on port 5000. So now you just need to send a request to 127.0.0.0:5000.

2. Netcat

Next way to see how your request is look like is netcat. It’s high featured tool sometimes called Swiss-army knife for TCP/IP. Here’s how to get raw request:

 nc -l 5000

Now netcat will redirect everything it gets to output so you will be able to check if you are doing everything right. The same way, just send you requests to 127.0.0.0:5000.

If you don’t have netcat and/or Flask installed you can do this this way on Fedora or Centos:

yum install python-flask

or

yum install nc

Dealing with high load average on Linux server

High load average is an issue which is familiar to almost all server owners who have popular sites and a lot of traffic. Usually it indicates that server can’t properly handle visitors’ requests. This short article aims to describe several simple items to consider while analyzing high load average on your Linux server.

1. MySQL slow query log

For highload projects it’s always better to keep MySQL slow query log enabled to analyze. It could be done this way.

[mysqld]
slow_query_log
slow_query_log_file=/var/log/mysqld-slow.log
long-query-time = 1
log-queries-not-using-indexes
log-warnings
expire_logs_days=7
[mysqld_safe]
log-error=/var/log/mysqld.log

Make sure MySQL has permissions to write slow log. Also note that by MySQL server doesn’t rotate slow log itself. So you would need to do it. It could consume a lot of gigabytes. It’s convenient to analyze slow log with use of mysqlsla tool.

2. Analyzing slow queries

To analyze the slow queries it’s possible to use SQL operator EXPLAIN. It allows to get information on how MySQL server performs the query, if it uses indexes and so on. Here’s an example:

mysql> explain SELECT 1 FROM `archive`.`log` LIMIT 1;

3. MySQL settings, InnoDB, MyISAM

MyISAM and InnoDB are the most often used MySQL storage engines. There are too many differences between them to describe here. But one the most important one is that MyISAM doesn’t support transactions while InnoDB does. Use mysqltuner.pl to get information on MySQL settings and follow the optimization tips. Make sure you understand parameters meaning before changing. If you don’t you’d better ask someone who understands. First optimization tune could be increasing key buffer pool and InnoDB buffer pool. MyISAM keeps in memory only indexes while InnoDB ties to keep the data along with the indexes. So InnoDB would perform the best if its data fits into memory (i.e. if you have 4G of InnoDB data, set innodb_buffer_pool at least to 5-6G). Besides you can use mysqreport and innotop to get more information on what is going on.

4. Swap, IO wait

If your server swaps a lot it will be a one of the sources of performance issues. Since traditional HDD are much slower than RAM the processes could wait for a quite long time to get a response. You can check swap using with top or free commands. One of the most important parameters here is “IO wait”. Here is a sample output of top:

Cpu(s): 12.1%us, 1.8%sy, 0.0%ni, 85.1%id, 0.3%wa, 0.5%hi, 0.2%si, 0.0%st
Mem: 8166644k total, 7773496k used, 393148k free, 165328k buffers
Swap: 8387572k total, 259428k used, 8128144k free, 5404860k cached

If its value is more then 30-40% you might want to optimize your server and/or add more RAM.

5. Apache and nginx

During last several years nginx has gained a high popularity. More and more users decide to use nginx instead of Apache. Although Apache is still one of the most popular web servers and the most featurer one. Since nginx uses epoll for working with the sockets states and file AIO to serve static files, often it performs much better then Apache. So if your application can be deployed with nginx you might want to consider the migration. At present almost all popular CMS and frameworks can be deployed with nginx: Django, Yii, WordPress and so on. Be aware that nginx doesn’t support .htaccess files so if you have them you will need to rewrite them for nginx. Fortunately there are free online converters which can help you with this.

6. Unnecessary services

If don’t need some services it would be better to disable them. It also better from security point of view. On Centos server you can use chkconfig to disable the service. On latest version of Fedora it can be done by systemctl. If you don’t use NFS you can disable all related services and so on.

7. Use performance monitoring software to track the metrics.

There are a lot of both self-hosted and web solutions to track the performance metrics. The graphics allows to analyze the dynamics much easier.