Japanese Characters

What's the general ettiquette on neoseeker in regards to Japanese characters in FAQs?

I've got a few FAQs I want to post here which use some Katakana, which would be fine if FAQs could be uploaded using ASCII compatible character sets like Shift-JIS or UTF-8. These are great because any viewers/editors which don't support these encodings still display Latin characters correctly.

However, I gather that FAQs HAVE to be in ASCII format. Is there some alternative I can use like the VFAQ thingies to embed Japanese characters? Or is there someone I can have a chat with about this issue?

I think we already have a number of Shift-JIS FAQs which we display without problems. I don't have a link handy right now, but your best bet is to send the faqs to faqs@neoseeker.com then post here, and after we update them we can all take a look.

From my memory though, I believe we can support Shift-JIS without problems at all, and we definitely 100% already have FAQs that have Shift-JIS and other characters in them.

They should also work correctly for formatted FAQs if you have them.

I know vicrabb has both formatted FAQs and regular FAQs that contain french accents in them and we've had to tweak bugs related to those a few times.

Update:

I found one FAQ as an example:

http://faqs.neoseeker.com/Games/PS2/valkyrie_profile_2_skill.txt

Unfortunately you have to ask the browser to go into SHIFT_JIS mode to see the japanese correctly. Now that I think about it I think this is an area we need to still work on, but we have no rules against sending us the SHIFT_JIS file, and if users tell their browser to switch modes then they can see the characters correctly.

Thanks for the ideas, they're much appreciated! You're the first person to take a technical interest in this so its actually quite exciting . I looked into this a few years ago, and though I'm no expert on character encodings, the major challenge we found in what you describe is detecting the character encoding of a text file, there doesn't seem to be any reliable way of doing it.

quote BoneIdol

Alternatively, you could just default to UTF-8 (via editing your server's mime types list) and tell people to send foreign language FAQs as UTF-8, since it can display just about every character in every alphabet in common usage today and won't screw over your existing ASCII files (although some of the foreign language FAQs may need converting).

Switching to UTF-8 is also difficult due the huge catalogue of existing iso-8859 and windows-1252 encoded FAQs we already have which contain accented characters (these seem to corrupt when told to output as UTF-8).

For me, multilanguage support is always a "supply chain" problem:

1. Asking users to encode files a certain way is unreliable because some might not follow directions
2. Many authors ask us to acquire their FAQs from other sites, which may have the files encoded differently than we expect.
3. Users who upload FAQs via the FAQCP might upload the wrong or different encoding
4. We need to know what encoding the source files were made with to output them properly
5. Detection of encoding is difficult and unreliable so we might output the wrong encoding headers.
6. Existing files have unknown encodings - an automated conversion may not be reliable due to the above note about unreliable encoding detection.
7. The system saving the files must not corrupt the file encoding being saved.
8. The server has to know to output the correct headers to the browser declaring the correct encoding.

I know GameFAQs appears to have solved the problem, so hopefully we can too. I'll take a look into this again more indepth, but it seems to me the holy grail is the ability to detect encoding of a file rather than ask people to upload in a specific encoding, so if you have any good ideas on doing so in PHP that would be "half the battle" right there!

(actually the true holy grail is if Apache itself could detect the encoding of a text file and output the correct headers at the time of serving plain txt files).

Extra note about encoding detection

If I understand character encoding properly the MAJOR problem is that there's no real way to detect encoding by reading the contents of a file. UTF-8 for instance is notoriously difficult to detect reliably. The technical reason appears to be that encoding itself is nothing more than a mapping of bytes to characters, so encoding detection relies on detecting sequences of bytes that may be exclusive to each encoding map. I suspect many encodings actually have significant overlap in byte sequences but map them to different characters, so the end result is that searching for byte sequences can be misleading depending on the range of maps you have to work with. Even detecting iso-8859 vs utf-8 is not trivial depending on the sequence of characters in the string because sometimes a specific string sequence in iso-8859 has bytes that map exactly to another sequence of characters in utf-8 (I think that's why you get so many bug reports and gotchas in the PHP mb_detect_encoding function. Other UTF-8 detection methods rely on reading the BOM, which is also unreliable because the BOM is optionally added or not added depending on the editor being used (I have text editors that when told to save in UTF-8 will save BOM and some that won't).