Fixing Garbled Japanese Files in Claude Code — Identifying Shift-JIS vs UTF-8 and Converting Safely

I am Hirokawa, an indie developer who has been shipping mobile apps since 1997 as a teenager and full-time since 2014, with cumulative downloads now past 50 million. A surprising amount of that workload involves Japanese CSVs and legacy Windows logs that predate UTF-8. When I started piping those files into Claude Code, the Read tool would return rows of 縺ゅ＞縺ゅ＞ and 繧定ｪｭ縺ｿ instead of readable text.

The cause is almost always encoding. Claude Code reads files as UTF-8, so Shift-JIS, CP932, EUC-JP, and BOM-prefixed UTF-8 files all corrupt at the Read step. Overwriting with the Edit tool while the file is still being misread is the worst thing you can do — it will destroy the original bytes. The reliable workflow is: detect, convert to a UTF-8 copy, verify, then read. This article walks through exactly that, with the commands I keep in my own claude.md.

Recognizing the garble — what each pattern usually means

You can often guess the original encoding just by looking at the broken output. A few common patterns:

Strings dominated by 縺, 繧, ｦ such as 縺ゅ＞縺ゅ＞ → originally UTF-8 misread as Shift-JIS, or the inverse. The original bytes are usually still intact.
Sequences of half-width kana dots like ・ｱ・ｱ・ｱ → originally Shift-JIS read as UTF-8. Very common in terminals.
First three bytes are EF BB BF → UTF-8 with BOM. Most editors hide it, but CSV parsers and shell scripts often choke on the invisible prefix.
ASCII text is fine but Japanese characters collapse → multibyte boundary corruption.

I have a folder of release notes for an older app that is still saved as EUC-JP. Once I added a ### Encoding sanity check section to that project's claude.md, the time-to-recover dropped from "half a coffee" to under a minute.

A one-minute detection routine

These commands cover almost every Japanese file you will encounter in the wild.

# 1. Quick check using file (macOS / Linux built-in)
file -bi notes_legacy.csv
# Possible outputs:
#   text/plain; charset=utf-8
#   text/plain; charset=iso-8859-1   ← classic "unknown 8bit", usually Shift-JIS
#   text/plain; charset=us-ascii
 
# 2. nkf -g, the Japanese-aware detector (brew install nkf / apt install nkf)
nkf -g notes_legacy.csv
# Possible outputs:
#   Shift_JIS
#   EUC-JP
#   UTF-8
 
# 3. Look at raw header bytes (the most reliable BOM check)
head -c 16 notes_legacy.csv | hexdump -C
# Leading EF BB BF → UTF-8 with BOM
# Leading FF FE     → UTF-16 LE
# Leading FE FF     → UTF-16 BE

In practice, when file -bi reports iso-8859-1, the file is Shift-JIS about 90% of the time. When it reports unknown-8bit, EUC-JP is still in play, so I reach for nkf -g for a second opinion. If nkf is not installed on the machine Claude Code is running on, install it with brew install nkf on macOS or apt install nkf on Linux.

Convert to UTF-8 before handing the file to the Read tool

Always produce a UTF-8 copy under a new name instead of overwriting the original. Keeping the original around means a botched conversion is a one-second rollback, not a "where is the backup" emergency.

# Shift-JIS → UTF-8 (the most common case)
iconv -f CP932 -t UTF-8 notes_legacy.csv > notes_legacy.utf8.csv
 
# Why CP932 instead of Shift_JIS: Windows-origin "Shift-JIS" files are technically
# CP932 (Microsoft's extended variant). Specifying Shift_JIS will fail on platform
# extensions like ①, ㈱, and 髙 that are common in Japanese business data.
 
# EUC-JP → UTF-8
iconv -f EUC-JP -t UTF-8 old_release_notes.txt > old_release_notes.utf8.txt
 
# Strip the UTF-8 BOM (when CSV parsers stumble on the leading prefix)
sed -i.bak '1s/^\xEF\xBB\xBF//' data.csv
 
# Add //TRANSLIT//IGNORE to tolerate untranslatable characters
iconv -f CP932 -t UTF-8//TRANSLIT//IGNORE notes_legacy.csv > notes_legacy.utf8.csv

Almost every unknown character error from iconv comes from platform-dependent characters: enclosed numerals, Roman numerals, archaic kanji forms. Adding //TRANSLIT//IGNORE quietly drops anything that cannot be represented, which is usually the right call for legacy business CSVs.

Before pointing Claude Code's Read tool at the new file, verify by eye.

head -3 notes_legacy.utf8.csv
file -bi notes_legacy.utf8.csv  # expect: charset=utf-8

Batch conversion workflow for Claude Code

When several files need converting, I drive it from a single Bash call. Writing this with concise output makes it easy for Claude to summarize.

# Convert every Shift-JIS or EUC-JP file under .legacy/ into .utf8/
mkdir -p .utf8
for f in .legacy/*.csv; do
  out=".utf8/$(basename "$f")"
  enc=$(nkf -g "$f")
  case "$enc" in
    "Shift_JIS")
      iconv -f CP932 -t UTF-8//TRANSLIT "$f" > "$out" && echo "OK   sjis  $f"
      ;;
    "EUC-JP")
      iconv -f EUC-JP -t UTF-8//TRANSLIT "$f" > "$out" && echo "OK   eucjp $f"
      ;;
    "UTF-8"|"BINARY")
      cp "$f" "$out" && echo "SKIP utf8  $f"
      ;;
    *)
      echo "WARN $enc $f"
      ;;
  esac
done

I ran this script over roughly 120 category-management CSV files for my wallpaper app, which had Shift-JIS and UTF-8 mixed together after years of edits across Windows and macOS. The whole archive was clean by morning, and asking Claude Code to dedupe categories under .utf8/ worked on the first try.

Four more pitfalls to rule out before you give up

Git's core.autocrlf is interfering: A Windows-side true paired with a macOS / Linux false can desync both newlines and encoding. Place a .gitattributes at the project root with *.csv text working-tree-encoding=CP932 eol=crlf to let Git handle the conversion automatically.
The CSV delimiter is \t or ;: If Japanese itself looks fine but the columns are mangled, the issue may not be encoding at all. head -1 to inspect the first row.
Claude Code's Bash environment expects UTF-8: Without LANG=ja_JP.UTF-8, even echo output can garble. Run locale to check, and if needed export LANG=ja_JP.UTF-8 LC_ALL=ja_JP.UTF-8.
An editor is re-adding the BOM: Check that VSCode is not configured to "Save with BOM", or pin it down with .editorconfig's charset = utf-8.

I once spent an hour on an App Store Connect revenue CSV that arrived in CP932, fighting pandas.read_csv ParserError until I finally ran file -bi and realized I was reading it as the wrong encoding the whole time. Making file -bi the first command saved me from repeating that mistake.

What to try next

When you hit garbled Japanese in Claude Code, run through these four steps in order:

file -bi <file> to fingerprint the encoding
If unsure, nkf -g for a second opinion
iconv -f CP932 -t UTF-8//TRANSLIT//IGNORE to produce a UTF-8 copy under a new name
Sanity-check with head and file -bi before pointing the Read tool at it

Once legacy files become safe to load, the range of work Claude Code can do over old Japanese assets — analysis, refactoring, migration — opens up considerably. I hope this saves you some of the time I burned figuring it out.