How to Read MySQL Binary Log Files (BinLog) with mysqlbinlog?

MySQL database server generates binary log files for every transaction to the databases, provided administrator does not disable or comment out the “log-bin” parameter in my.cny configuration file. The binary log files are written in binary format. Although the binary logs, or also known as logbin are mainly used for MySQL database replication purpose, sometimes you may need to examine or read the contents of binary logs in text format, where the mysqlbinlog utility will come in handy.

Binary log file, which normally has name with the format host_name-bin.xxxxxx and will be store in your data directory, could not be opened and read straight away as it’s in unreadable binary format. To read the binary logs in text format, we can make use of mysqlbinlog command, which also able to readrelay log files written by a slave server in a replication setup. Relay logs have the same format as binary log files.

To use mysqlbinlog utility is simple, simply use the following command syntax to invoke mysqlbinlog after login in as root (else you have to specify user name and password) to shell via SSH:

mysqlbinlog [options] log_file ...

So to read and display the contents of the binary log file named binlog.000001, use this command:

mysqlbinlog binlog.000001

The binary log files and its data are likely to be very huge, thus making it almost impossible to read anything on screen. However, you can pipe the output of mysqlbinlog into a file which can be open up for later browsing in text editor, by using the following command:

mysqlbinlog binlog.000001 > filename.txt

To reduce the amount of data retrieved from binary logs, there are several options that can be used to limit the data that is been returned. Among the useful ones are listed below:

–start-datetime=datetime

Start reading the binary log at the first event having a timestamp equal to or later than the datetime argument. The datetime value is relative to the local time zone on the machine where you run mysqlbinlog. The value should be in a format accepted for the DATETIME or TIMESTAMP data types. For example:

mysqlbinlog --start-datetime="2005-12-25 11:25:56" binlog.000001

–stop-datetime=datetime

Stop reading the binary log at the first event having a timestamp equal or posterior to the datetime argument. This option is useful for point-in-time recovery. See the description of the –start-datetime option for information about the datetime value.

–start-position=N

Start reading the binary log at the first event having a position equal to the N argument. This option applies to the first log file named on the command line.

–stop-position=N

Stop reading the binary log at the first event having a position equal or greater than the N argument. This option applies to the last log file named on the command line.




For more info, you can visit http://dev.mysql.com/doc/refman/5.0/en/mysqlbinlog.html




Thanks!

Kuldeep

Show Real IP in Apache Access Log (CDN)

Recently we have got some DDoS attacks on our FIREWALL and we want to find out the Source IP from where we were getting these attacks. 

Since we are using CDN(Content Delivery Network) and maximum of static contents are being served from CDN. So we will not be able get source IPs in our logs and we need to see the real IP to make sure we can do some log processing accurately.
The Level3 CDN basically working like example below:

Level 3 Edge Server will work as the reverse-proxy to cache the website contents from requests they received on their edge servers anywhere around the world. This will speed up the website loading page and decrease the server load of the web server. The only problem of this implementation is that you can only seen Level3,s IP connecting to your web server.

As the web server has no information about the source IP unless the edge server (CDN provider) gives it a header on the HTTP request to specify it. Then we have asked LEVEL3 to verify if they are sending a header like this. TRUE-CLIENT-IP is a header that is send on LEVEL3’s network.
Once we come to know the header (if exists) that is sent by level3 then we have added below LogFormat to our httpd.conf file.
  If want to implement this for particular domain, then implement this on that particular domain.
----
LogFormat "%{TRUE-CLIENT-IP}i %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
----
 After this you need to restart Web-Server to make changes effective.This will result that each line in your access log will now begin with the TRUE-CLIENT-IP. (Of course replace TRUE-CLIENT-IP with the header that is sent on level3 network.
Hope this will help!!

Thanks!!