Bandwidth monitoring: new features at ServerBeep.com

Recently we have added the new features. Now we can describe them in a more detailed way.

One of the most popular features is network bandwidth monitoring for Linux servers. To use it you will need to install our free Python daemon which will collect the network usage statistics and push them to our servers (we provide the installation guide.) As soon as we have enough data you will be able to monitor your server(s) bandwidth. The daemon sends the updates every 5 minutes at the moment. To get the data the daemon reads /proc pseudo filesystem so it should not lead to any performance issues even on high-traffic servers.

At the moment only graphical (MRTG-like) information is available. We provide daily and weekly graphs. Now we support Linux only version of the daemon. If you need to monitor FreeBSD or Windows servers please let us know.

Along  with bandwidth monitoring, the daemon can collect data on memory, CPU, disk space usage and load average. So, almost all items which are necessary to monitor for getting smoothly running Linux server. Of course, website and service monitoring features are still available for free.

Register now to start using bandwidth and performance monitoring!

Linux server monitoring is out

network usage monitoring

Hello! Earlier we announced our new set of features. Today we have pushed the updates into production for Linux server monitoring at ServerBeep.com. So now along with website and services availability we provide monitoring of:

  • CPU usage,
  • memory usage,
  • load average,
  • disk space usage,
  • and bandwidth usage.

You can find the screenshots and the description here.

We are very pleased to offer you server monitoring along with website and service availability checks. Our monitoring features will allow you to track your servers performance trough time. Now there are accessible daily and weekly graphs.

Getting started

To start using free Linux server monitoring you need to register and to setup the daemon on your Linux box. The installation process is simple and quick. We provide short but full instruction for the installation. The daemon is configurable and allows to disable some of features if you don’t need them.

The daemon is written in pure Python and doesn’t require any third-party software. It was tested on Centos and Fedora servers with Python versions 2.6 and 2.7, but also should work on other operation systems with these versions of Python.

The daemon was developed for high-traffic servers and shouldn’t lead to any performance issues.  In the most cases we read only /proc filesystem.

How the daemon works

Once in 5 minutes the daemon checks the server parameters and sends them to us. After that it sleeps until the next check. The daemon is open sourced.

Register for free now!

We are always glad to receive any of your input, so if you have any suggestions or questions please contact us.

Free Linux server performance monitoring by ServerBeep.com

Hi!

During last few months we were working on a set of new features. And we are glad to announce them now.

What

In the nearest future you will be able to monitor:

  • CPU usage,
  • memory usage,
  • load average,
  • disk space usage,
  • and bandwidth usage.

on your Linux dedicated or virtual server.

Why

  • It allows to analyze your Linux box performance and the dynamics.
  • It could help to find out the issues. (For instance, if you network usage reduces dramatically it could be related to some issues with the application or the server.)
  • The monitoring provides the possibility of performance and scaling issues forecasting. (For example, it could be CPU or memory related issues. If you have load average more then 50, for instance, it could mean that it’s time to analyze where it comes from.)
  • Our free monitoring service is the first place to check what is going on with your server.

How

To start using you would need to install our Python daemon which would collect and send the data to our servers. The daemon doesn’t need any special libraries and should work from the box (still in testing stage, so we’ll update the requirements soon). The daemon will be open sourced so you could check how it works.

Why CPU, memory, load average, disk and network usage monitoring?

Because from out point of view they are the most important and vital parameters for the Linux performance analysis. Even brief view on a dashboard could help to find the bottlenecks. Historical data allows easily inspect the server performance during certain period of time.

With ServerBeep everything you need to start monitor your server performance is to get an authentication key for the server in the membership area, download the daemon and configure it. Once it starts sending us the data you will be able to monitor it. The whole process should only take a few minutes.

Why Linux?

Linux is one of the most popular server operation systems and we use it ourselves. So the first step in performance server monitoring is Linux. If you need Windows or FreeBSD performance monitoring please kindly let us now.

Register now and start using free Linux server performance monitoring!

Email infrastructure made right. Part 3: feedback loops.

In the previous posts (part 1, part 2) we covered some required items which should be implemented to send emails safely. Here’s one more item you should consider before starting sending emails: feedback loops. It works next way. You register on FBL and if some of your recipients mark your mail as spam you will get a special message about it. So you can take necessary actions. You can get a list of available feedback loops on corresponding Wikipedia page. It’s free of charge. For MSN, Zoho and Yahoo you will need their accounts, Google doesn’t have FBL.

What you can monitor with ServerBeep.com

ServerBeep.com is a free web monitoring service.  Here’s a brief overview of our key features and a list of items you can monitor with our service.

Websites

If you run a website and you want to know when it goes down you can easily use ServerBeep.com for this. Just register, add your website  and assign the contacts for the site. And if your site becomes inaccessible due to various reason we will send you an email notification. And when it comes back we will send the notification once again.

VPS and dedicated servers

Our features list includes all common services checks: HTTP, FTP, mail services, MySQL and SSH. So you are able to monitor the most important services of your VPS or a dedicated server. You are free to add up to 10 services per account now. You can also monitor your services availability.

Launching Python applications with Gunicorn and systemd

At ServerBeep.com we love Python and use it widely. There are a lot of ways to deploy Python application. One of them is using Gunicorn.

Gunicorn is a Python WSGI HTTP Server for UNIX. So if you have WSGI complaint application you can run it with Gunicorn. It can be, for instance, Django or Flask powered applications.

From the other hand, last versions of Fedora Linux ship with systemd. Systemd is a featured replacement of old SysV system. Among other features there is one which allows to launch the processes after server boot. To use systemd for the applications called app launch create file /lib/systemd/system/gunicorn-app.service:

[Unit]
Description=gunicorn-app

[Service]
ExecStart=/usr/bin/gunicorn -D -n gunicorn-app -w5 tracwsgi:application -b 127.0.0.1:8000 --access-logfile /home/app/log/app.serverbeep.net-access.log --error-logfile /home/app/log/app.serverbeep.net-error.log
Type=forking
User=app
Group=app
Restart=always
StandardOutput=syslog
StandardError=syslog
WorkingDirectory = /home/app/

[Install]
WantedBy=multi-user.target

And enable it:

systemctl enable gunicorn-app.service

Now after server boots you will get your application run. After you start it Gunicorn will handle the requests which are sent to 127.0.0.1:8000. Next step could be nginx or Apache setup as reverse proxy. It means that webserver listens well-known port 80 and serves some files (usually static ones) directly and send rest of requests to Gunicorn. It responses back and webserver provides a reply to a user.

With systemd you don’t have to setup supervisord or other processes monitoring and management tools. Systemd will take care of it. It’s easy and neat way to launch the daemons.

Email infrastructure made right. Part 2: handling bounces.

In our previous post on email infrastructure we were talking about minimum requirements which should be implemented. Today we will cover bounces handling. What do the bounces mean? In short words, it’s a non delivery report. You could get if you send a mail to non-existing user (error code 5.1.1 ) or a user which has full mailbox (5.2.2).

There are kinds of the bounces:

  • soft bounces;
  • hard bounces.

If you get soft bounces it means that you can try to send an email later. And hard bounces mean that email account you are trying to send to is invalid. Such email accounts should not be used for further mail sending. And if you have a lot of subscribers you need a way to get rid of invalid email accounts. But how can you do it? Fortunately there’s a simple way. Probably you’ve heard about VERP. It stands for Variable envelope return path. Here’s how it works (we will cover Sendmail and Dovecot based setup on Centos).

Let’s assume we want to send a mail to joe@example.com from no-reply@serverbeep.com. If there’s no user joe@example.com we will get a bounce message to no-reply@serverbeep.com. But why don’t we collect all bounces in one mailbox? For simplicity of handling bounces we can setup some mail account. Let’s called it bounces@serverbeep.com and setup our LDA to deliver all mails to some user account (it could be system user bounces or some other account depending on your setup). Set up Sendmail to forward all incoming mail sent to bounces@serverbeep.com to system account bounces. Open /etc/aliases files and add next line:

bounces@serverbeep.com: bounces

Tell Sendmail to use Dovecot as LDA:

FEATURE(`local_procmail', `/usr/libexec/dovecot/deliver',`/usr/libexec/dovecot/deliver -d $u')
MODIFY_MAILER_FLAGS(`LOCAL', `-f')
MAILER(procmail)dnl

It should be placed to /etc/mail/sendmail.mc.

Apply Sendmail changes:

cd /etc/mail ; make all restart

If you don’t have Dovecot installed it’s time to install it:

yum install dovecot

Here’s an example of Dovecot configuration file working as LDA:

# 2.0.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.1.1.el6.x86_64 x86_64 CentOS release 6.3 (Final)
disable_plaintext_auth = no
lda_mailbox_autocreate = yes
listen = 127.0.0.1
mail_location = maildir:~/Maildir
mbox_write_locks = fcntl
passdb {
driver = pam
}
postmaster_address = postmaster@serverbeep.com
ssl = no
userdb {
driver = passwd
}
protocol lda {
auth_socket_path = /var/run/dovecot/auth-master
info_log_path = /var/log/dovecot/delivery-info.log
log_path = /var/log/dovecot/delivery.log
postmaster_address = postmaster@serverbeep.com
}

(Make sure you have /var/log/dovecot and dovecot has the permissions to write there).

Well, since we prepared everything let’s get back to VERP. To get a bounces while sending a mail to joe@example.com to bounces user maildir we need to add Return-Path header to our mail. Short example in PHP:

<?php
mail("joe@example.com", "Message subject", "Message body", "From: no-reply@serverbeep.com\n", "-fbounces+joe=example.com@serverbeep.com");
?>

So, there will be additional header in the email:

Return-Path: bounces+joe=example.com@serverbeep.com

In this case if there’s no user joe@example.com we will get a bounce message in our POP3/IMAP4 account bounces from where the messages could be easily fetched with simple Python script.

import imaplib
M = imaplib.IMAP4('127.0.0.1')
M.login('bounces', 'strongest_password')
M.select()
typ, data = M.search(None, 'ALL')
for num in data[0].split():
typ, data = M.fetch(num, '(RFC822)')
print 'Message %s\n%s\n' % (num, data[0][1])
M.close()
M.logout()

Once you get the messages you can parse it to get a list of invalid email accounts and make a modification of your users table in the database.

Using svn+ssh with non-standard port

If you use svn over SSH you may encounter an issue related to non-standard SSH port:

[dev@serverbeep web]$ svn checkout svn+ssh://dev@dev.devserver.com/home/svn-dev/
svn: To better debug SSH connection problems, remove the -q option from 'ssh' in the [tunnels] section of your Subversion configuration file.
svn: Network connection closed unexpectedly
[dev@serverbeep web]$

It’s because ssh client tries to connect to default SSH port 22. The solution is quite easy. You just need to add to your svn configuration file ~/.subversion/config next line:

ssh = $SVN_SSH ssh -q -p 2201

It should be in section [tunnels]. And this is it. Now you can checkout or update or whaterver you need to do with your remote SVN repository.

Email infrastructure made right. Part 1: basic requirements.

If you send emails quite often like we do at ServerBeep.com (we send email notifications if something bad happen with our customers’ websites or services), you had better to prepare your email setup before you start. It’s better to do it before than to solve the issues after. There are few items described below which should be consider as required before you start.

Check you IP addresses history

Sender Score and Sender Base allow you to check your IP history. If you find any issues you’d better consider changing it.

MX and PTR DNS records

To send a mail you have to setup PTR and MX DNS records properly so your IP address should point to your domain. A lot of SMTP server would reject your mail if you don’t have correct PTR record.

[root@sb1 ~]# host 88.198.20.28
28.20.198.88.in-addr.arpa domain name pointer serverbeep.com.
[root@sb1 ~]#

SPF

SPF stands for Sender Policy Framework. It allows you to tell other SMTP servers which IP and/or domain are allowed to send a mail from your domain. And it’s quite easy to setup. There’re a lot of online services which could help you with it.

Here’s for instance, our SPF record:

serverbeep.com. 42008 IN SPF "v=spf1 ip4:88.198.20.28 ip6:2a01:4f8:130:32a4::28 ~all"

Besides, it would be better to have the same one in TXT record. Just in case.

If you check the headers from the mail you received to your Gmail account you can see how Google treats SPF:

Received-SPF: pass (google.com: domain of no-reply@serverbeep.com designates 2a01:4f8:130:32a4::28 as permitted sender) client-ip=2a01:4f8:130:32a4::28;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of no-reply@serverbeep.com designates 2a01:4f8:130:32a4::28 as permitted sender) smtp.mail=no-reply@serverbeep.com; dkim=pass header.i=@serverbeep.com

Using Gmail is the simplest way to check if you set up SPF properly. Besides there are a lot of other ways to check online. Just google for ‘spf check’.

DomainKeys Identified Mail

To setup DKIM you need to generate public and private keys. Public should be added to your domain zone while private key should be securely stored on you server. SMTP server, Sendmail for instance, uses private key to add a digital signature which can be used for email validation.

Here’s an example of DKIM record:

default._domainkey.serverbeep.com. 41765 IN TXT "v=DKIM1\; k=rsa\; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDXFOEq2OCK1Bpp7YpId3nSgJX8fcfl5bx3a9aiDHm5nCAx0ZmAPacOoo+dmFlBfbcAIbi2BPons6w/uIW4rSeFAuybGw04/wMOkVit1OJPoiCsGW9BJLPeMnez+6m32zv3drVjgeywxtVDNbQphOIJdk4S88O2hlWOsusYv4sEdQIDAQAB"

The same as for SPF, you could check your DKIM by sending a mail to your Gmail account. Look for DKIM-Signature and Authentication-Results headers.

Necessary mail aliases: abuse@, postmaster@ and fbl@

If there are any issues other postmasters would send a mail to some of these aliases. So again, you should definitely have them set up.

Smart Network Data Services by Microsoft

Register on SNDS to be informed if there’re any issues with Hotmail.

Of course, it’s not all required steps. But probably the most important ones.

Debugging HTTP requests with Flask and netcat

ServerBeep.com is in ongoing development process. To meet your requirements we try to do our best to provide the best and the simplest website and monitoring service. This short post is dedicated to debbuging HTTP requests with Flask and netcat.

1. Flask

Ok, so we need to see what the webserver gets. With Flask it’s pretty simple. Flask is light but power Python framework built on top of Werkzeug and has a lot of extensions and features. It’s not so featured framework as Django but for simple applications it fits very well. (Before we were adepts of web.py and were quite happy with it. Now some of our internal applications are still powered with it, but Flask seems to have more active community and is under more dynamic development at the moment). This a complete code of how to get a request as Flask it sees:

from flask import Flask, request
app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])

def hello_world():
    from pprint import pprint
    pprint (request.form)
    # If you need headers just comment out next line. 
    # pprint (request.headers)
    return 'UP'

if __name__ == '__main__':
    app.debug = True
    app.run()

And this is it. By default Flask application waits for the requests on port 5000. So now you just need to send a request to 127.0.0.0:5000.

2. Netcat

Next way to see how your request is look like is netcat. It’s high featured tool sometimes called Swiss-army knife for TCP/IP. Here’s how to get raw request:

 nc -l 5000

Now netcat will redirect everything it gets to output so you will be able to check if you are doing everything right. The same way, just send you requests to 127.0.0.0:5000.

If you don’t have netcat and/or Flask installed you can do this this way on Fedora or Centos:

yum install python-flask

or

yum install nc