MySQL Fabric — Three Node High Availability Server Farm

So how do you use MySQL Fabric to set up a Highly Available Server Farm? The last two postings in this series were on installing Fabric on a master and then setting up slaves. Now it is time to get get the Fabric Farm started.

The Fabric controller is node number 1 and the slaves are 10, 20, and 30 at IP 10.10.13.1, 10, 20, and 30 respectively. I am keeping with the last octet of the IP addresses for clarity. I am trying to recreate the Fabric Farm from this image.

On the 10, 20, and 30 systems we need to add the following to the mysld section of the my.cnf
[mysqld]
binlog-format=ROW
log-slave-updates=true
gtid-mode=on
enforce-gtid-consistency=true
master-info-repository=TABLE
relay-log-info-repository=TABLE
sync-master-info=1
report-host=192.168.0.201
report-port=3306
server-id=1
log-bin=mysql1-bin.log

Next set up replication grants on the data nodes and the controller. We want the ability to use any of the slaves and the READ-WRITE master plus be able to switch them at will.
mysql> CHANGE MASTER TO MASTER_HOST='10.10.13.10', MASTER_USER='replication', MASTER_PASSWORD='*****', MASTER_AUTO_POSITION=1;
Query OK, 0 rows affected, 2 warnings (0.01 sec)

mysql> START SLAVE;
Query OK, 0 rows affected (0.00 sec) I started out with 10 as the replication master and used the above on 20 and 30.

A 5.7 Problem
The data farm machines are running MySQL 5.6 but my Fabric controller is running 5.7. If you are also running 5.7 note that by default your SQL MODE is strict and you will get
mysqlfabric manage setup
[INFO] 1423080020.902196 – MainThread – Initializing persister: user (Fabric), server (localhost:3306), database (fabric).
Error: Command (CREATE TABLE log (subject VARCHAR(40) NOT NULL, reported TIMESTAMP /*!50604 (6) */ NOT NULL, reporter VARCHAR(64) NOT NULL, message TEXT, category int NOT NULL, type int NOT NULL, INDEX key_subject_reported (subject, reported), INDEX key_reporter (reporter), INDEX key_reported (reported), INDEX key_category (category), INDEX key_type (type)), ()) failed accessing (localhost:3306). 1067 (42000): Invalid default value for ‘reported'
Opps! 5.7 has STRICT for SQL MODE as the default and I assumed reported was being fed no data. So it was time to set the default SQL MODE to ANSI and restart my Fabric controller’s instance.

So Create the Fabric Database already!
mysqlfabric manager setup creates the necessary tables in the Fabric state store. These tables will be used to store the topology information. The admin user created here can be used to create other users that can connect to fabric as can be seen below. And this command needs to be run only once.

mysqlfabric manage setup
[INFO] 1423080461.323271 - MainThread - Initializing persister: user (Fabric), server (localhost:3306), database (fabric).
Finishing initial setup
=======================
Password for admin user is not yet set.
Password for admin/xmlrpc:
Repeat Password:
Password set.
Password set.
No result returned


Side note 1: For over over a decade I have been typing the five letter m-y-s-q-l group alone for a good part of my professional workday and my fingers have a hard time adding f-a-b-r-i-c. Fellow old timers take care.
Side note 2: It can get confusion which password for which function is being asked for at any one time. I am sure more familiarity on my part would help but I can see folks getting confused easily.
Side note 3: In the early days of MySQL cluster I had to dig into the various pieces as it was not intuitive how things fit together and Fabric for me is the same way. I hope more time will on Fabric will build familiarity.

Orville and Wilbur Wright Time
mysqlfabric manage start &
[1] 12690
dstokes@dstokes-E7240:~/vagrant$ [INFO] 1423080590.563764 - MainThread - Initializing persister: user (Fabric), server (localhost:3306), database (fabric).
[INFO] 1423080590.571004 - MainThread - Loading Services.
[INFO] 1423080590.585766 - MainThread - MySQL-RPC protocol server started, listening on localhost:32275
[INFO] 1423080590.592161 - MainThread - Fabric node starting.
[INFO] 1423080590.596949 - MainThread - Starting Executor.
[INFO] 1423080590.597056 - MainThread - Setting 5 executor(s).
[INFO] 1423080590.597318 - Executor-0 - Started.
[INFO] 1423080590.597674 - Executor-1 - Started.
[INFO] 1423080590.597998 - Executor-2 - Started.
[INFO] 1423080590.598294 - Executor-3 - Started.
[INFO] 1423080590.599090 - Executor-4 - Started.
[INFO] 1423080590.599224 - MainThread - Executor started.
[INFO] 1423080590.607582 - MainThread - Starting failure detector.
[INFO] 1423080590.609208 - XML-RPC-Server - XML-RPC protocol server ('127.0.0.1', 32274) started.
[INFO] 1423080590.609764 - XML-RPC-Server - Setting 1 XML-RPC session(s).
[INFO] 1423080590.610071 - XML-RPC-Session-0 - Started XML-RPC-Session.


Now the three data nodes can be turned into a server farm.
mysqlfabric group create davesfarm
Password for admin:
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
------------------------------------ -------- ------- ------
cc54e1b2-4e54-4d02-99b4-3692d35fcce1 1 1 1

state success when description
----- ------- ------------- ------------------------------------------------------------------
3 2 1423080661.24 Triggered by .
4 2 1423080661.25 Executing action (_create_group).
5 2 1423080661.27 Executed action (_create_group).

This farm is called davesfarm by me but just 5ca1ab1e-a007-feed-f00d-cab3fe13249e by Fabric.

So what does the farm look like?
mysqlfabric group lookup_servers davesfarm
Password for admin:
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
----------- ------- ------ ---- ------


After the farm is created, it needs to be made active.

mysqlfabric group activate davesfarm
Password for admin:
[INFO] 1423081400.519033 - Executor-0 - Monitoring group (davesfarm).
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
------------------------------------ -------- ------- ------
5ac9a044-c650-428e-ba50-50a6718f2342 1 1 1

state success when description
----- ------- ------------- ------------------------------------------------------------------
3 2 1423081400.49 Triggered by .
4 2 1423081400.5 Executing action (_activate_group).
5 2 1423081400.53 Executed action (_activate_group).


Cool!

mysqlfabric group lookup_groups
Password for admin:
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

group_id description failure_detector master_uuid
--------- ----------- ---------------- -----------
davesfarm None 1 None

Now Add an admin
An admin is needed to populate the farm.
mysqlfabric user add client --user=admin
Password for admin:
Add a new Fabric user
=====================
Username: client
Protocol (default xmlrpc):
Password:
Repeat Password:

Select role(s) for new user
ID Role Name Description and Permissions
-- ---------- ---------------------------
1 superadmin Role for Administrative users
+ Full access to all core Fabric functionality
2 useradmin Role for users dealing with user administration
+ User administration
+ Role administration
3 connector Role for MySQL Connectors
+ Access to dump commands
+ Reporting to Fabric

Enter comma separated list of role IDs or names: 1
Fabric user added.
No result returned


And On This Farm He Had a …
group add davesfarm 10.10.13.10
Password for admin:
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
------------------------------------ -------- ------- ------
550b1ba6-082d-4fef-9808-8fdfd6397de9 1 1 1

state success when description
----- ------- ------------- ------------------------------------------------------------------
3 2 1423083005.22 Triggered by .
4 2 1423083005.23 Executing action (_add_server).
5 2 1423083005.27 Executed action (_add_server).

mysqlfabric group add davesfarm 10.10.13.20
Password for admin:
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
------------------------------------ -------- ------- ------
4fd59253-bd10-42b9-82cd-0d61959a695c 1 1 1

state success when description
----- ------- ------------- ------------------------------------------------------------------
3 2 1423083297.97 Triggered by .
4 2 1423083297.98 Executing action (_add_server).
5 2 1423083298.01 Executed action (_add_server).
mysqlfabric group add davesfarm 10.10.13.30
...


Welcome to Farmville!!!
mysqlfabric group lookup_servers davesfarm
Password for admin:
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
------------------------------------ ----------- --------- --------- ------
0977d639-abbb-11e4-a804-0800273b9c80 10.10.13.30 SECONDARY READ_ONLY 1.0
1f96a235-abb9-11e4-a7f8-0800273b9c80 10.10.13.10 SECONDARY READ_ONLY 1.0
304e20cd-abba-11e4-a7ff-0800273b9c80 10.10.13.20 SECONDARY READ_ONLY 1.0

So now the farm is populated but we need to create a pecking order by making one of them the primary.

mysqlfabric group promote davesfarm --slave_id=1f96a235-abb9-11e4-a7f8-0800273b9c80
Password for admin:
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

uuid finished success result
------------------------------------ -------- ------- ------
af79f3e0-fda2-4531-8aab-c8a48ad036a6 1 1 1

state success when description
----- ------- ------------- ------------------------------------------------------------------
3 2 1423084103.73 Triggered by .
4 2 1423084103.73 Executing action (_define_ha_operation).
5 2 1423084103.75 Executed action (_define_ha_operation).
3 2 1423084103.74 Triggered by .
4 2 1423084103.75 Executing action (_check_candidate_fail).
5 2 1423084103.77 Executed action (_check_candidate_fail).
3 2 1423084103.77 Triggered by .
4 2 1423084103.77 Executing action (_wait_slave_fail).
5 2 1423084103.81 Executed action (_wait_slave_fail).
3 2 1423084103.8 Triggered by .
4 2 1423084103.81 Executing action (_change_to_candidate).
5 2 1423084103.93 Executed action (_change_to_candidate).

mysqlfabric group lookup_servers davesfarm
Password for admin:
Fabric UUID: 5ca1ab1e-a007-feed-f00d-cab3fe13249e
Time-To-Live: 1

server_uuid address status mode weight
------------------------------------ ----------- --------- ---------- ------
0977d639-abbb-11e4-a804-0800273b9c80 10.10.13.30 SECONDARY READ_ONLY 1.0
1f96a235-abb9-11e4-a7f8-0800273b9c80 10.10.13.10 PRIMARY READ_WRITE 1.0
304e20cd-abba-11e4-a7ff-0800273b9c80 10.10.13.20 SECONDARY READ_ONLY 1.0


So Fabric has been started, a farm has been established, and it is ready for testing. But the testing will have to wait for another post.

And if you want to stop Fabric, the command is:
shell> mysqlfabric manage stop

Some blogs to help:

• vnwrites.blogspot.com/2014/11/tutorial-introduction-to-fabric-groups.html
• blog.marceloaltmann.com/mysql-fabric-installing-instalar/
• http://www.clusterdb.com/mysql-fabric/mysql-fabric-adding-high-availability-to-mysql