MySQL Master-Master Replication, Heartbeat, DRBD, Apache, PHP, Varnish MegaHOWTO

October 8th, 2014

I created this HOWTO while building a new development environment today. The intention is to take a single Apache2/Varnish/MySQL environment and scale it to two servers, with one effectively a “hot-standby” – increase redundancy and continuity whilst maintaining current performance. This HOWTO is based on Linux Debian-76-wheezy-64-minimal 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64

Our current server has IP 192.168.201.1/24 and our new server has IP 192.168.201.7.

Section #1: Set up MySQL Master/Master Replication


First, we’ll set up MySQL master to master replication. In this configuration, data can be written and read from either host. Bear in mind that issues may exist with autoincrement fields when written to at the same time. There are other caveats with replication so ensure to research them along with how to deal with corruption and repair before considering this setup for a live application. Also be sure to be using the same version of MySQL on both servers – this may not always be necessary, however unless you are very familiar with any changes between versions, not doing so could spell disaster.

Read the rest of this entry »

Ethernet over mains power lines

July 26th, 2010

I’ve been using a really clever device for the last few years that a lot of people seem to be unaware exists. It’s an ethernet over powerlines adapter – one such example is the Devolo dLAN. In a nutshell, you plug it into the mains, and connect the Ethernet socket to your network device. You can then plug as many others as you like to various other powerpoints and extend your network wherever the power stretches. Devolo do ones that run up to 200mbit. It’s a theoretical maximum, although I’ve got 177mbit before which is impressive. It has a couple of downsides:

1. It won’t traverse 3 phase power. I’ve tried it, and I’ve ended up with a very weak/nonexistant signal which is probably more inductance than anything else.
2. Obviously it doesn’t handle bad cables well – it doesn’t much like extension cables either.
3. Different circuits work about as well as 3 phase power, the only signal you will get is probably inductance between the two circuits.

Some advantages:
1. It travels pretty far. I’ve had over 150mbit between adapters at opposite ends of the house.
2. No new cabling
3. Fully supports standard Ethernet so all network protocols will work just fine over it.
4. I love it

For anyone running a home or office network and not fortunate to have Ethernet points cabled in, I strongly recommend these devices, you’ll never know the difference.

Linux and the Huawei HSDPA 3G E220 modem for mobile broadband

February 28th, 2009

Installing the Huawei E220 modem under Linux is so so easy. Probably easier than running through the Windows XP Setup tool actually!

Prerequisites:

1. Kernel version equal to or greater than 2.6.20 (There are workarounds availble for older kernels)
2. wvdial (You can use any serial dialer I guess)
Read the rest of this entry »

DNS based Load Balancing

January 22nd, 2009

There are two main options for DNS based load balancing. The first and most simple is the round robin option. We can use this for ‘A (alias) records’ and ‘MX (Mail-eXchanger) records’.

We can specify a priority for MX records. If we specify the same priority for multiple MX records, the querying client will toss a coin and ‘randomly’ decide which to use. The same applies to A records. This should provide with a reasonable split between your various records however provides no mechanism for server loads or using any kind of intelligence to route queries.

Another option is to return a record based on intelligence. Assume we are trying to balance load between web servers. The two popular methods we can use are to return a record based on knowledge of the load of the web servers, or alternatively return a record based on originating IP (location) of the requesting client.

This is all well and good however there are a number of considerations, specifically that DNS was not intended to be operated in this way.

  1. You can set your records expire time to as low as you like, it will still be cached in circumstances by the browser and/or the resolver. This method will not account for ‘downed’ or ‘overloaded’ servers, they will still receive traffic.
  2. Due to caching, should your browser or resolver hold on to the record, it will blindly access the same IP next time the host name is requested, without requerying the DNS server and ignorant of the changed network conditions.

Linux IP Address Configuration Static or DHCP

January 18th, 2009

It’s simple really..

You can define a static IP as follows:

ifconfig <interface> <ipaddress> netmask <mask> broadcast <broadcast>
e.g. ifconfig eth0 192.168.1.2 netmask 255.255.255.0 broadcast 192.168.1.255

You can then define a route out to the internet:

route add default gw <router>
Which has the same effect as: route add -net 0.0.0.0/0 <router>
i.e. route add default gw 192.168.1.1

Or if you’d like to acquire an address via a local DHCP server:

dhclient <interface>
e.g. dhclient eth0

That’s all there is to it! Now.. these settings won’t stick on reboot, you’ll have to refer to your distro’s startup files. On debian you want /etc/network/interfaces

A sample stanza for the above configuration:

auto eth0
iface eth0 inet static
address 192.168.1.2
netmask 255.255.255.0
broadcast 192.168.1.255
gateway 192.168.1.1

All done!

Simple POP3 Communication HOWTO

December 16th, 2008

POP3 is an incredibly simple protocol, and with the most basic commands, you can access your POP3 server ‘by hand’ with this POP3 HOWTO without the need for a client. You can find the entire POP3 RFC here http://www.ietf.org/rfc/rfc1939.txt

Now, down to business. I have created a temporary test account:  test@iodigitalsec.com – please don’t try and access this as by the time you see this, it’s already been removed! I’ll use telnet to access the service, and send simple plain text commands. I’ve sent myself a test email, which I will also retrieve and then delete. Conversation as follows, I have highlighted my own commands in bold:

Read the rest of this entry »

A BIND9 zonefile and commentary

December 15th, 2008

I’m often asked for a copy of various zone files for Bind, that other users may use as a template. Here’s the zonefile for www.adampalmer.me/iodigitalsec:

$TTL 604
@       IN      SOA      iodigitalsec.com. root.iodigitalsec.com. (
2008101023        ; Serial
172800         ; Refresh
900         ; Retry
1209600         ; Expire
3600 )       ; Negative Cache TTL
;
IN      NS      ns3.apnichosting.com.
IN      NS      ns2.apnichosting.com.
IN      MX      10      mail3.sasdataservices.com.
IN      MX      100     mail2.sasdataservices.com.
IN      MX      1000    backup-0.l3.iodigitalsec.com.
IN      A       217.10.156.197
*                       CNAME   iodigitalsec.com.

I’ll now cover each type of record briefly, and explain the ellusive decimal point.

The SOA or “start of authority” record indicates the domain name “iodigitalsec.com” and the email address of the domain administrator “root@iodigitalsec.com”, replacing the at symbol with a decimal point (this decimal point does not have the same meaning as those later on). There is only one SOA record allowed per domain. Contained within the SOA record is also a serial number, refresh, retry, expiry and TTL. The serial number is the ‘version’ of the zone. This is generally incremented each time the zone is updated. The refresh is used by the slave or secondary DNS server as an instruction on how often to update in seconds. The ‘retry’ is the length in seconds that the slave DNS server should wait before retrying to contact an unreachable primary DNS server. The expiry specifies how long until the slave DNS server stops responding to requests for this domain name, should the primary DNS server remain unreachable. If the primary DNS server becomes available again, the timer is reset. Lastly, the Negative TTL or ‘time to live’ value indicates how long the server will cache a NAME ERROR (NXDOMAIN) record. The longest permitted is 3h (10800 seconds).

On to the more simple records…
Read the rest of this entry »

rsync over SSH, SSH key login, public keys, automated backups

October 24th, 2008

This tutorial will cover how to set up a simple backup job between two machines using rsync and ssh. You will need HOST A and HOST B, whereby HOST B is your target backup service.

On HOST B:

ssh-keygen -t rsa  # Press enter to accept the default options.
mv ~/.ssh/id_rsa ~/.ssh/identity
cat ~/.ssh/id_rsa.pub

Read the rest of this entry »

Linux on a Mikrotik 532a , Part 5 Final – OpenWRT and Custom Scripts

October 19th, 2008

Follow on from: http://www.adampalmer.me/iodigitalsec/linux-on-a-mikrotik-532a-part-4-customization-debian-scripts-shaping-firewall-nat-picolcd/

I’ve used OpenWrt previously to this project to build some firmwards for the Linksys Router WRT54 range. OpenWrt is an incredibly powerful and small Linux distro. Although debian is probably better suited to the reasonably powerful hardware, I wanted to give OpenWrt a go anyway.

Unless you’re running a MIPS 4Kc processor on your host which I’m guessing you’re not, you’ll either need to cross compile your binaries, or just compile them natively on the device itself. Compiling on the device works fine as long as you have the relevant packages, however if I was going to build a 2.6 kernel, I’d rather do it on an x86 quad core intel host, rather than waiting a week for the device to do it. I also wanted to minimize the writes on the CF card.

OpenWrt comes with a nice buildroot environment which you can read about and download from www.openwrt.org using Subversion.

Here http://downloads.openwrt.org/kamikaze/docs/openwrt.html#x1-310002 is a great HOWTO on getting the build root environment set up on your x86 host.

Also, see: http://wiki.mikrotik.com/wiki/RB500_Linux_SDK – this is a very complete HOWTO, which is why I’ve not covered most of the installation process and just detailed customizations.

You’ll need to select the RB5xx target for the kernel. Also, run:

make kernel_config

In your build root top directory, and add USB support (as my one is modded for USB which is not RB5xx default.

While you’re there, browse to the networking options and make sure you have everything you want, specifically the schedulers for traffic shaping.

Read the rest of this entry »

Linux on a Mikrotik 532a, Part 4 – Customization, Debian Scripts, Shaping, Firewall, NAT, picoLCD

October 11th, 2008

Follow On From: 05 Oct 08 APNIC Box – Linux on a Mikrotik 532a, Part 3 – Installing Debian, Prebuilt Disk Image

Following on from the previous article, I’ve written some scripts which you’ll find in the /root/scripts/ directory of the prebuilt image. I’ve attached and commented them here, as they could also be useful elsewhere.

bridge.sh #For setting up a simple bridge
Read the rest of this entry »

APNIC Box – Linux on a Mikrotik 532a, Part 3 – Installing Debian, Prebuilt Disk Image

October 5th, 2008

Follow on from 01 Oct 08 APNIC Box – Linux on a Mikrotik 532a, Part 2

The device runs a 2.4.30 kernel on a debian woody (mipsel) environment. If anyone can contribute anything for 2.6.x and debian etch, that would be great.

Installation instructions:

Read the rest of this entry »

APNIC Box – Linux on a Mikrotik 532a, Part 2 – Hardware Modifications

October 1st, 2008

Follow on from 01 Oct 08 APNIC Box – Linux on a Mikrotik 532a, Part 1

Custom Hardware Modifications

Here’s a labelled image of the inside of the device. You can also look towards the bottom left of the image for my simple solder modifications. Enlarge the image to see the labels.

APNIC Box Image 2

APNIC Box Image 2

1. External 2.4GHz/5GHz antenna. Same on opposite side.
2. 5V solder point
3. 5V connector for miniPCI USB card
4. 2x 2USB Headers. 1 Header in use providing 2x USB interfaces, one to regular host connector for mass storage or other usb connection. Other port for picoLCD on top
5. 512MB CF card
6. miniPCI USB controller

On the underside of the board there is a single miniPCI socket which houses an Atheros 5212 802.11a/b/g miniPCI card. It has two antenna outputs which run under the board and two the two external antennae. I haven’t taken a picture of this but if anyone really wants to see it, I will power down the device, get a picture of it and post it here.

Read the rest of this entry »

APNIC Box – Linux on a Mikrotik 532a, Part 1 – The Device

October 1st, 2008

I put this device together for fun sometime around the start of 2007. The ideas that spawned this was using OpenWRT on a Linksys WRT54G access point. A surprisingly powerful and full linux distro with all kinds of advanced capabilities running on a Linksys wireless router which I’d previously thought to be a reasonably dumb device with computing power more comparable to a calculator than a PC. The project opened my eyes to embedded devices, and I wondered what device base I should start with. To cut a long story short and for reasons that I can’t even remember anymore I came across the Mikrotik Routerboard 532A and decided that I should start with that.

Conception

APNIC Box Image 1

APNIC Box Image 1

Here’s a picture of the device from the outside with some labels, view the full image to see them.

1. Status LEDs. Blue at the bottom left shows it’s on, orange at the top right shows that there’s wifi activity.
2. Ethernet (eth0)
3. Standard Serial Console (57600, 8 N 1)
4. Ethernet (eth1)
5. Ethernet (eth2)

Read the rest of this entry »

Wireless Hacking, Problems with WEP, Wireless Security and WPA

September 22nd, 2008

Unfortunately today there are still a huge range of wireless OEM equipment being shipped with WEP as standard. WEP has been known as vulnerable for a long time. This HOWTO assumes Linux familiarity, compatible hardware, the ability to read and troubleshoot, and a brain.

Hacking your wireless network is not difficult, and here’s a procedure you can use to test:

You’ll need:
1. A PC and wireless network.
2. A linux PC/laptop with a wireless networking device

Method:
1. Boot your (debian) pc
2. wget http://download.aircrack-ng.org/aircrack-ng-1.0-rc1.tar.gz
3. tar -xzf aircrack-ng-1.0-rc1.tar.gz
4. cd aircrack-ng-1.0-rc1
5. ./configure
6. make
7. make install

Read the rest of this entry »

95th percentile billing explaination

September 17th, 2008

95th percentile billing commonly misspelled as “percential” is a method used by some NOCs to change for bandwidth.

The system is simple and essentially discards the top 5% of your traffic peaks, and then uses the next value down as your bandwidth rate. 5% of a month is 36 hours. This might sound like a bit of a scam, because you’re being billed for bandwidth consumption that you may not have used, but it’s not difficult to get it to work for you.

If you’re hosting a site where a lot of content is downloaded, it may be better to go for bandwidth billing. A client’s content server uses about 8,000GB transfer per month and shows a 95th percentile of 34mbit/sec. It’s certainly cheaper to pay for 8,000GB transfer over 30+mbit/sec dedicated.
Read the rest of this entry »