MySQL Fabric : Testing High Availability Fail Over


Tine to test MySQL Fabric’s High Availability fail over. Last entry I set up a three node Fabric HA farm. This time I am going to ‘pull the plug’ on the servers in the farm and see if Fabric will sense the loss of the server and gracefully keep working.

To sum up the setup, I am using a laptop as the Fabric Controller and three virtual servers for the farm. The virtual servers are Vagrant instances

Example of Fabric environment tested

MySQL Fabric HA Diagram — Nice illustration of tested environment

creatively named db1 at 10.10.13.10, db2 at 10.10.13.20, and db3 at 10.10.13.30. All the MySQL instances are 5.6.23 and the MySQL Utilities are 1.6.1-alpha. See the last blog entry for details.

The MySQL Fabric HA farm has been started.
mysqlfabric group lookup_servers myfarm
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
------------------------------------ ----------- --------- ---------- ------
09d13be1-cdd0-11e4-8643-0800273b9c80 10.10.13.10 SECONDARY READ_ONLY 1.0
0f611996-cdd0-11e4-8643-0800273b9c80 10.10.13.20 SECONDARY READ_ONLY 1.0
11aae7e7-cdd0-11e4-8643-0800273b9c80 10.10.13.30 PRIMARY READ_WRITE 1.0
Number ’30 is the PRIMARY node. A simple test program’s connector queries the farm and is actually talking to ’30. So time to issue vagrant down db3 to power off that server.

With db3 shut down, the test program now is communicating with ’20. So far, so very good.

mysqlfabric group lookup_servers myfarm
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
------------------------------------ ----------- --------- ---------- ------
09d13be1-cdd0-11e4-8643-0800273b9c80 10.10.13.10 SECONDARY READ_ONLY 1.0
0f611996-cdd0-11e4-8643-0800273b9c80 10.10.13.20 PRIMARY READ_WRITE 1.0
11aae7e7-cdd0-11e4-8643-0800273b9c80 10.10.13.30 FAULTY READ_WRITE 1.0

MySQL Fabric has gracefully promoted ’20 from SECONDARY to PRIMARY. Time to bring down ’20!

mysqlfabric group lookup_servers myfarm
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
------------------------------------ ----------- ------- ---------- ------
09d13be1-cdd0-11e4-8643-0800273b9c80 10.10.13.10 PRIMARY READ_WRITE 1.0
0f611996-cdd0-11e4-8643-0800273b9c80 10.10.13.20 FAULTY READ_WRITE 1.0
11aae7e7-cdd0-11e4-8643-0800273b9c80 10.10.13.30 FAULTY READ_WRITE 1.0
And the test program is chatting with ’10.

MySQL Fabric gracefully handles the failure of two out of the three nodes in the High Availability farm.

And for the pedantic, shutting off the last server in the farm will generate a ‘Connection with Fabric failed’ message.

This entry was posted on Friday, April 3rd, 2015 at 8:09 am and is filed under Administration, MySQL Fabric. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

One Response to MySQL Fabric : Testing High Availability Fail Over

  1. MySQL Fabric – Faulty to Secondary | Open Source DBA's Blog says:
    April 10, 2015 at 8:50 am

    […] Fabric is a great tool for managing a farm of MySQL servers. In the last two posts you can see how High Availability Fail over and Sharding work. When things are working right, the group lookup_servers will look something like […]

    Reply

Leave a reply to MySQL Fabric – Faulty to Secondary | Open Source DBA's Blog Cancel reply