mysql5.1→mysql5.6へアップデート後、勝手にDBダウン

2015年05月06日に実施した、mysqlのバージョンアップ作業に伴い、そのまま放置していたんですが、1ヶ月過ぎたあたりから、勝手にＤＢがダウンしてしまい、wordpressにアクセスすると「データベース接続確立エラー」となることが多くなった。

これまではそれ程頻度が高く無かったんで、気がついた時にmysqlを起動しなおすとしていたんですが、ここ最近はすぐに落ちるようになってしまったんで、暫定対応ではなく恒久的対応をしないと面度臭くなったという感じです。

ググルと沢山出てくるんですが、我が家の場合、既に稼動されている状況からの接続確立エラーで、wordpressの設定も変更していない状況から、上記のmysqlのアップグレードに伴った影響だと思われました。5.1から5.6系への変更がそんなに大きな変化があるなんて、調べもしないで実施したんで、まあ、当然の結果でもあるんですが・・・

で、滅多に見ないmysql.logを確認すると、エラーが出まくっているではありませんか！

いくつか問題があって、一つづつやっつけていきました。

１．
2015-07-01 22:47:26 10230 [ERROR] Native table 'performance_schema'.'setup_actors' has the wrong structure
2015-07-01 22:47:26 10230 [ERROR] Native table 'performance_schema'.'setup_objects' has the wrong structure
2015-07-01 22:47:26 10230 [ERROR] Native table 'performance_schema'.'table_io_waits_summary_by_index_usage' has the wrong structure
2015-07-01 22:47:26 10230 [ERROR] Native table 'performance_schema'.'table_io_waits_summary_by_table' has the wrong structure
2015-07-02 21:31:05 7fde64477700 InnoDB: Error: Table "mysql"."innodb_table_stats" not found.

こんなようなパフォーマンス関係のテーブルでエラーがドッサリと出る。

とりあえず、アップグレードしたときに以下のデータベースのアップグレードをしていなかったので、実行しました。全ての項目がOKになることを確認して、この問題はとりあえず完了としました。
mysql_upgrade -u root -p

２．
InnoDB: Error: Table "mysql"."innodb_table_stats" not found.

これは、以下のバグとのことで、記載の通り実施して解決。
/*
temporary fix for problem with windows installer for MySQL 5.6.10 on Windows 7 machines.
I did the procedure on a clean installed MySql, and it worked for me, at least it stopped
lines of innodb errors in the log and the use of transient innodb tables. So, do it at
your own risk..

1. drop these tables from mysql:
innodb_index_stats
innodb_table_stats
slave_master_info
slave_relay_log_info
slave_worker_info

2. delete all .frm & .ibd of the tables above.

3. run this file to recreate the tables above (source five-tables.sql).

4. restart mysqld.

Cheers,
CNL
*/

CREATE TABLE `innodb_index_stats` (
`database_name` varchar(64) COLLATE utf8_bin NOT NULL,
`table_name` varchar(64) COLLATE utf8_bin NOT NULL,
`index_name` varchar(64) COLLATE utf8_bin NOT NULL,
`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`stat_name` varchar(64) COLLATE utf8_bin NOT NULL,
`stat_value` bigint(20) unsigned NOT NULL,
`sample_size` bigint(20) unsigned DEFAULT NULL,
`stat_description` varchar(1024) COLLATE utf8_bin NOT NULL,
PRIMARY KEY (`database_name`,`table_name`,`index_name`,`stat_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0;

CREATE TABLE `innodb_table_stats` (
`database_name` varchar(64) COLLATE utf8_bin NOT NULL,
`table_name` varchar(64) COLLATE utf8_bin NOT NULL,
`last_update` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
`n_rows` bigint(20) unsigned NOT NULL,
`clustered_index_size` bigint(20) unsigned NOT NULL,
`sum_of_other_index_sizes` bigint(20) unsigned NOT NULL,
PRIMARY KEY (`database_name`,`table_name`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin STATS_PERSISTENT=0;

CREATE TABLE `slave_master_info` (
`Number_of_lines` int(10) unsigned NOT NULL COMMENT 'Number of lines in the file.',
`Master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the master binary log currently being read from the master.',
`Master_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The master log position of the last read event.',
`Host` char(64) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT 'The host name of the master.',
`User_name` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The user name used to connect to the master.',
`User_password` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The password used to connect to the master.',
`Port` int(10) unsigned NOT NULL COMMENT 'The network port used to connect to the master.',
`Connect_retry` int(10) unsigned NOT NULL COMMENT 'The period (in seconds) that the slave will wait before trying to reconnect to the master.',
`Enabled_ssl` tinyint(1) NOT NULL COMMENT 'Indicates whether the server supports SSL connections.',
`Ssl_ca` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The file used for the Certificate Authority (CA) certificate.',
`Ssl_capath` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The path to the Certificate Authority (CA) certificates.',
`Ssl_cert` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the SSL certificate file.',
`Ssl_cipher` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the cipher in use for the SSL connection.',
`Ssl_key` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the SSL key file.',
`Ssl_verify_server_cert` tinyint(1) NOT NULL COMMENT 'Whether to verify the server certificate.',
`Heartbeat` float NOT NULL,
`Bind` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'Displays which interface is employed when connecting to the MySQL server',
`Ignored_server_ids` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The number of server IDs to be ignored, followed by the actual server IDs',
`Uuid` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The master server uuid.',
`Retry_count` bigint(20) unsigned NOT NULL COMMENT 'Number of reconnect attempts, to the master, before giving up.',
`Ssl_crl` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The file used for the Certificate Revocation List (CRL)',
`Ssl_crlpath` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The path used for Certificate Revocation List (CRL) files',
`Enabled_auto_position` tinyint(1) NOT NULL COMMENT 'Indicates whether GTIDs will be used to retrieve events from the master.',
PRIMARY KEY (`Host`,`Port`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Master Information';

CREATE TABLE `slave_relay_log_info` (
`Number_of_lines` int(10) unsigned NOT NULL COMMENT 'Number of lines in the file or rows in the table. Used to version table definitions.',
`Relay_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the current relay log file.',
`Relay_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The relay log position of the last executed event.',
`Master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the master binary log file from which the events in the relay log file were read.',
`Master_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The master log position of the last executed event.',
`Sql_delay` int(11) NOT NULL COMMENT 'The number of seconds that the slave must lag behind the master.',
`Number_of_workers` int(10) unsigned NOT NULL,
`Id` int(10) unsigned NOT NULL COMMENT 'Internal Id that uniquely identifies this record.',
PRIMARY KEY (`Id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Relay Log Information';

CREATE TABLE `slave_worker_info` (
`Id` int(10) unsigned NOT NULL,
`Relay_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`Relay_log_pos` bigint(20) unsigned NOT NULL,
`Master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`Master_log_pos` bigint(20) unsigned NOT NULL,
`Checkpoint_relay_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`Checkpoint_relay_log_pos` bigint(20) unsigned NOT NULL,
`Checkpoint_master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
`Checkpoint_master_log_pos` bigint(20) unsigned NOT NULL,
`Checkpoint_seqno` int(10) unsigned NOT NULL,
`Checkpoint_group_size` int(10) unsigned NOT NULL,
`Checkpoint_group_bitmap` blob NOT NULL,
PRIMARY KEY (`Id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Worker Information';

３．
2015/7/6に落ちる現象あり。ログを調べると
2015-07-06 10:44:53 3765 [Note] InnoDB: Initializing buffer pool, size = 128.0M
InnoDB: mmap(137363456 bytes) failed; errno 12
2015-07-06 10:44:53 3765 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2015-07-06 10:44:53 3765 [ERROR] Plugin 'InnoDB' init function returned error.
2015-07-06 10:44:53 3765 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2015-07-06 10:44:53 3765 [ERROR] Unknown/unsupported storage engine: InnoDB
2015-07-06 10:44:53 3765 [ERROR] Aborting

と、エラーが出ている。調べると、swapサイズが足りないようだったんで、スワップファイルを新規に作成して追加した。暫く稼動させていないと、上記のエラーが発生するかわからないので、暫く様子見。
[root@blog ~]# swapon -s
Filename Type Size Used Priority
/dev/sda5 partition 4193276 0 -1
/swapfile file 1048572 0 -2


これで、ログを確認してエラーログが出ていないことを確認した。長時間稼動させても、これで勝手に落ちたりしないことを祈るばかり。

この対応でも落ちるようだと、何かしらの外部からの攻撃によって落ちたと思うしかないかな？！
まあ、ひとつづつ、ノンビリと解決できればいいや。