MySQL Fabric : Testing High Availability Fail Over


Tine to test MySQL Fabric’s High Availability fail over. Last entry I set up a three node Fabric HA farm. This time I am going to ‘pull the plug’ on the servers in the farm and see if Fabric will sense the loss of the server and gracefully keep working.

To sum up the setup, I am using a laptop as the Fabric Controller and three virtual servers for the farm. The virtual servers are Vagrant instances

Example of Fabric environment tested

MySQL Fabric HA Diagram — Nice illustration of tested environment

creatively named db1 at 10.10.13.10, db2 at 10.10.13.20, and db3 at 10.10.13.30. All the MySQL instances are 5.6.23 and the MySQL Utilities are 1.6.1-alpha. See the last blog entry for details.

The MySQL Fabric HA farm has been started.
mysqlfabric group lookup_servers myfarm
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
------------------------------------ ----------- --------- ---------- ------
09d13be1-cdd0-11e4-8643-0800273b9c80 10.10.13.10 SECONDARY READ_ONLY 1.0
0f611996-cdd0-11e4-8643-0800273b9c80 10.10.13.20 SECONDARY READ_ONLY 1.0
11aae7e7-cdd0-11e4-8643-0800273b9c80 10.10.13.30 PRIMARY READ_WRITE 1.0

Number ’30 is the PRIMARY node. A simple test program’s connector queries the farm and is actually talking to ’30. So time to issue vagrant down db3 to power off that server.

With db3 shut down, the test program now is communicating with ’20. So far, so very good.

mysqlfabric group lookup_servers myfarm
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
------------------------------------ ----------- --------- ---------- ------
09d13be1-cdd0-11e4-8643-0800273b9c80 10.10.13.10 SECONDARY READ_ONLY 1.0
0f611996-cdd0-11e4-8643-0800273b9c80 10.10.13.20 PRIMARY READ_WRITE 1.0
11aae7e7-cdd0-11e4-8643-0800273b9c80 10.10.13.30 FAULTY READ_WRITE 1.0

MySQL Fabric has gracefully promoted ’20 from SECONDARY to PRIMARY. Time to bring down ’20!

mysqlfabric group lookup_servers myfarm
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
------------------------------------ ----------- ------- ---------- ------
09d13be1-cdd0-11e4-8643-0800273b9c80 10.10.13.10 PRIMARY READ_WRITE 1.0
0f611996-cdd0-11e4-8643-0800273b9c80 10.10.13.20 FAULTY READ_WRITE 1.0
11aae7e7-cdd0-11e4-8643-0800273b9c80 10.10.13.30 FAULTY READ_WRITE 1.0

And the test program is chatting with ’10.

MySQL Fabric gracefully handles the failure of two out of the three nodes in the High Availability farm.

And for the pedantic, shutting off the last server in the farm will generate a ‘Connection with Fabric failed’ message.

One Response to MySQL Fabric : Testing High Availability Fail Over

  1. […] Fabric is a great tool for managing a farm of MySQL servers. In the last two posts you can see how High Availability Fail over and Sharding work. When things are working right, the group lookup_servers will look something like […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s