New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DB Upgrade script 2009100701.sql corrupts UTF8/multibyte characters in database in some cases (Trac #1928) #1928
Comments
brettp wrote on 40171732-08-16 I have tested this with Japanese, Korean, and Russian on 4 different servers (including Ubuntu 9.10 with defaults) and cannot duplicate. Cash - You said 9.10--Was this a fully updated copy or a fresh install from their ISO? Also, did you just use
to install or specify specific versions (mysql-server-5.0)? |
cash wrote on 40171853-08-23 Two people reported it in the community site so far. I did an install from the ISO and then did a dist-upgrade. I did not specify a specific mysql version for the install. I'll grab all these details tonight and add them to the ticket. |
trac user kako wrote on 40172443-05-16 This occurs on UTF-8 format, on Spanish versión loose all special character (ñ, í, ó) and cannot login. I've just install on debian etch server with mysql-server-5.0. Thx. |
brettp wrote on 40172690-09-02 Could someone please post the result of running
both before and after a faulty migration? |
brettp wrote on 40172691-08-15 Also, what the string should be in UTF8 would be useful too :) |
cash wrote on 40172813-07-23 Okay - I figured out how I did this when trying to reproduce the reports. I entered the data as UTF8 and the conversion assumes latin1 and the ending result was the corruption. I was in too much of a hurry. You can throw out my data point. This gives us three reports. Two gave absolutely no details. The third: kako posted a screengrab of the website on the community site that has latin1 characters coming through. This probably means the new code was copied over but the upgrade script was not run. That leaves two reports with no details... I think we can close this and anyone who runs into the problem can reopen. |
brettp wrote on 40172860-02-21 Agreed. Closing this one. One thing though--If users trying to upgrade modified core (as suggested and warned against in this thread [svn:1]) the upgrade will result in corrupted databases. If so, the users will need to restore from a backup, delete the 1.7 engine/schema/upgrades/2009100701.sql migration, and run the upgrade again. [svn:1] http://community.elgg.org/mod/groups/topicposts.php?topic=236843&group_guid=16#annotation-432444 |
trac user kako wrote on 40173793-05-28 Replying to brettp:
|
trac user kiang wrote on 40189130-03-10 I met the same problem and the solution to remove '2009100701.sql' worked for me. I haven't had tried to modify the core of elgg. But our website is using charset utf-8 ( http://oss.tw/elgg/ ) and the tables were already in utf-8. Maybe I did create the database with utf-8 and the tables inherited when installing elgg 1.6.1. I think the upgrade script could detect if the table is using utf-8 charset by sending the query 'SHOW TABLE STATUS' to the database. Ref: http://dev.mysql.com/doc/refman/5.0/en/show-table-status.html |
brettp wrote on 40189316-04-29 It doesn't matter what charset the tables are in this case; it matters what charset PHP uses to communicate with MySQL. 1.7 forward uses "SET NAMES utf8" to tell PHP to community with the MySQL server in utf8. Previous versions used the default of latin1. I don't know of a way to test what charset previous clients used. Simply testing what charset the tables are in isn't enough. After some research, I believe there may be a way to test for this. kiang, can you please do the following if you are able? Use a standard mysql command line client (PHP My Admin won't work) to connect to your server and select your Elgg database, then run these queries:
I would be very interested to see the output of this on your server. Thanks. |
trac user kiang wrote on 40189328-02-11 Sure, here you are. ;)
|
brettp wrote on 40189338-04-15 kiang Thanks for the quick reply! Is this an upgraded 1.7 database or a previous one? |
trac user kiang wrote on 40189354-03-23 brettp, this is the clone of the database before upgrading(1.6.1). |
Title changed from |
Milestone changed to |
Attachment added by brettp on 40209766-08-26: utf8_test.php |
brettp wrote on 40209777-12-07 For everyone having this problem, could you please download the attached 'utf8_test.php' file, place it in your Elgg root, run it from a web browser, and copy the output here? This is a simple file that outputs information about your default PHP settings that I hope can be used to conditionally run this upgrade. |
trac user gatchi wrote on 40241771-01-27 I have a same problem with korean language. i think it is exactly same problem with Sam. Result with the 'uft8_test.php' test on my server is this array(2) { ["Variable_name"]=> string(20) "character_set_client" ["Value"]=> string(4) "utf8" } array(2) { ["Variable_name"]=> string(24) "character_set_connection" ["Value"]=> string(4) " utf8" } array(2) { ["Variable_name"]=> string(22) "character_set_database" ["Value"]=> string(4) "utf8" } array(2) { ["Variable_name"]=> string(24) "character_set_filesystem" ["Value"]=> string(6) "binary" } array(2) { ["Variable_name"]=> string(21) "character_set_results" ["Value"]=> string(4) "utf8" } array(2) { ["Variable_name"]=> string(20) "character_set_server" ["Value"]=> string(4) "utf8" } array(2) { ["Variable_name"]=> string(20) "character_set_system" ["Value"]=> string(4) "utf8" } array(2) { ["Variable_name"]=> string(18) "character_sets_dir" ["Value"]=> string(38) "/usr/local/mysql/share/mysql/charsets/" } |
brettp wrote on 40248891-09-21 (In [svn:5565]) Fixes #1928: UTF8 migration moved from db to code to detect if it is required. Bumped version to 1.7.1 (2010033101). |
brettp wrote on 40248905-02-05 Thanks to the feedback from the utf8 test I was able to write a new upgrade that I believe resolves this problem. Please grab the latest 1.7 branch and give it a try and reopen if there are any problems. I'd like to have this tested on a variety of servers before releasing 1.7.1. Thanks. |
Original ticket http://trac.elgg.org/ticket/1928 on 40171649-06-10 by cash, assigned to unknown.
Elgg version: 1.7
I'm creating a ticket in case other users have this problem.
To reproduce:
The text was updated successfully, but these errors were encountered: