Non-utf-8 form data decoded incorrectly


I am working on a legacy app running on ColdFusion and I am using Lucee to try to run it
locally and understand it.

The app is using a Japanese encoding(windows-31j / shift_jis) for html and form post data.
windows31-j is an extension to ascii. The extended characters are 2 bytes long.
The first byte is always greater than 0x80, but the second byte can be in the range
of normal ascii.

To escape such characters, the browser (checked with edge and firefox) uses
the “%” escape notation for only those bytes that need escaping. This seems
to cause a problem with Lucee.

When both bytes are escaped, such as “%96%ee” the sequence is correctly decoded.
But when the second byte is not escaped, such as “%8eq”, which might have been
encoded as “%8e%71”, the sequence become converted to a bogus character.

I searched through the Lucee source, and I think URLDecoder.decode() is doing the conversion.
It seems that the function assumes that when % is used, then the remaining bytes that belong
to the extended character are all escaped with % as well.

I think this is a bug. Should I open an issue tracker entry?


Don’t forget to tell us about your stack!

Using docker image lucee/lucee:

OS: the one that comes with the docker image
Java Version: jdk11
Tomcat Version: 9.0
Lucee Version:

Can you show us some short code example that does that escaping?

always please check with the latest version first, 5.3.9 is rather old


Sorry about the old version I am using. It takes a little effort to replace it.
However I looked at the source of the 6.0 branch and I think the code in that branch (still) has this problem.

Shown below is a standalone class that demonstrates the proper decoding.
(I couldn’t upload as an attachment since my account here is new.)

Note SJIS and Windows-31J are roughly equal.


public class Codec {

    public static byte[] utf8_to_sjis_bytes(String s) throws UnsupportedEncodingException {
        return s.getBytes("Windows-31J");

    public static String sjis_bytes_to_utf8(byte b[]) throws UnsupportedEncodingException {
        return new String(b, "Windows-31J");

    public static String escape_like_a_browser_does(byte[] bytes) {
        StringBuffer sb = new StringBuffer();
        for (int i = 0; i < bytes.length; i++) {
            // escape only the non-ascii bytes.
            if ((bytes[i] & 0x80) != 0) {
                sb.append("%" + Integer.toHexString(bytes[i] & 0xff));
            } else {
                sb.append((char) bytes[i]);
        return sb.toString();

    // Compare this from URLDecoder.decode() in Lucee
    public static byte[] decode_escaped_str(String s) {
        byte[] buf = new byte[s.length()];
        int pos = 0;
        for (int i = 0; i < s.length(); i++) {
            if (s.charAt(i) == '%') {
                int code = Integer.parseInt(s.substring(i + 1, i + 3), 16);
                buf[pos++] = (byte) code;
                i += 2;
            } else {
                buf[pos++] = (byte) s.charAt(i);
        byte[] result = new byte[pos];
        for (int i = 0; i < pos; i++) {
            result[i] = buf[i];
        return result;

    public static void demonstrate(String s1) throws UnsupportedEncodingException {
        byte s1_sjis_bytes[] = utf8_to_sjis_bytes(s1);
        String escaped_s1 = escape_like_a_browser_does(s1_sjis_bytes);
        byte[] decoded_s1_bytes = decode_escaped_str(escaped_s1);
        String decoded_s1 = sjis_bytes_to_utf8(decoded_s1_bytes);
        System.out.println(s1 + " -> " + escaped_s1 + " -> " + decoded_s1);

    public static void main(String args[]) {
        try {
            demonstrate("矢"); // 0x96,0xee in Shift_JIS
            demonstrate("子"); // 0x8e,0x71 in Shift_JIS
        } catch (UnsupportedEncodingException e) {

When it is run, here is the console output

矢 -> %96%ee -> 矢
子 -> %8eq -> 子

In the code I implied that the java strings are utf-8, but I think it was UTF-16. Sorry about that.

I made a fix to as follows on the 6.0 branch and confirmed it works.
You have to wait until you go through the whole string before feeding it to the String
constructor with encodings specified.

	public static String decode(String s, String enc, boolean force) throws UnsupportedEncodingException {
		if (!force && !ReqRspUtil.needDecoding(s)) return s;

		boolean needToChange = false;
		int numChars = s.length();
		byte bytes[] = new byte[numChars];
		int pos = 0;
		int i = 0;

		while (i < numChars) {
			char c = s.charAt(i);
			switch (c) {
			case '+':
				bytes[pos++] = (byte) ' ';
				needToChange = true;
			case '%':

				try {
					while (((i + 2) < numChars) && (c == '%')) {
						needToChange = true;
						bytes[pos++] = (byte) Integer.parseInt(s.substring(i + 1, i + 3), 16);
						i += 3;
						if (i < numChars) c = s.charAt(i);

					if ((i < numChars) && (c == '%')) {
						needToChange = true;
						bytes[pos++] = (byte) c;
				catch (NumberFormatException e) {
					needToChange = true;
					bytes[pos++] = (byte) s.charAt(i);
					bytes[pos++] = (byte) s.charAt(i + 1);
					bytes[pos++] = (byte) s.charAt(i + 2);
					i += 3;
				needToChange = true;
				bytes[pos++] = (byte) c;

		return (needToChange ? new String(bytes, 0, pos, enc) : s);

Here is a sample form page.
Save it from your unicode capable text editor specifying the encoding to be windows-31j.
Store it as webapps/ROOT/index.cfm

On current 6.0 when you submit the form, the string marked as ‘bad’ will be corrupted.

<cfcontent type="text/html; charset=Windows-31J">
<cfset setEncoding("Form", "Windows-31J")>

<head><title>Form encoding test</title></head>
<h1>Form encoding test</h1>
On the lucee admin app, set Settings/Language/Compiler/Template charset to windows-31j<br>
Store this file as webapps/ROOT/index.cfm using windows-31j as an encoding.

<cfparam name="good" default="矢">
<cfparam name="bad" default="子">

<form action="index.cfm" method="post">
<label for="good">good : </label>
<input type="text" name="good" value="#good#"><br>

<label for="bad">bad : </label>
<input type="text" name="bad" value="#bad#"><br>

<input type="submit" name="button" value="送信">

<cfif isdefined("form.good")>
<p>good : #form.good#</p>
<p>bad : #form.bad#</p>
<p>button label(bad): #form.button#</p>



I appreciate if this gets fixed on 6.0 soon, but there are (surprisingly) a lot of PRs ongoing.
I could explain more, or move on and submit an issue on Jira, or submit a PR.

Which is preferred?

A PR with tests would be great, so please go ahead and create an issue in jira and then submit a PR linked to the issue

ok. I’ll try.

I digged a little deep on this and found out that the problem is a bit far reaching.

In lucee code, there is the URLDecoder class as well as the URLDecode() function.

By fixing URLDecoder, I can fix the issue on both HTTP POST payloads and query parameters on the URL. However, to also cover the CFML URLDecode function, which can be called from CFML code, you have to fix URLDecode() also.

URLDecoder implements its own decoding code, but URLDecode() hands the job to And that code has the same problem. It is not just Lucee.

In the wider java community this issue has been overcome by the URLCodec class in Apache commons codec.

So based on my findings my suggestion would be, to replace the implementation inside Lucee of both the URLDecoder class and the URLDecode() functions with the URLCodec class.

Does that sound OK? I am slightly concerned about breaking someone’s
application that relied on the behavior or in some
obscure way.

Suggestions welcome.


I’m all for adopting mature upstream libraries, Lucee/Ralio is so old that a lot of these didn’t exist back in the day.

As you rightly point out, we are beholden to existing behaviour and ACF quirks, but it’s worth trying

That’s why we have test cases :wink:

I’d say go for it and see how it goes!


OK. I will try.

Hi. I ran the modification through the test suite, and it failed.

I found out that the test case for the URLDecode() function requires that a malformed input is bypassed silently.


	assertEquals("%&/", "#URLDecode('%&/')#");

This line exists from the creation of the file in the git repo.

To satisfy this test case, the current URLDecode() implementation first gives a try. It will throw an exception on such input. Then to give the string a second chance it is passed to Lucee’s URLDecoder() class, which is permissive for such input.

To keep this test passing, you have to keep the custom implementation of the URLDecoder class.

I don’t think that this particular test case was added to confirm that a required feature works. I guess that it was added merely to check that the salvation logic for malformed input works as the author of URLDecoder() intended.

Since I don’t have Adobe ColdFusion at my hand, I can’t confirm the behavior of ACF.

Is it OK to remove that line from the test case?

I have another question.

How can I test java classes that are not directly exposed as CF functions?

The URLDecoder class ( that I want to fix is used from Lucee when form post data is converted and passed to the CFML application.

I don’t see any test code, at least in the Lucee repo, doing such tests.

Is there a place for such kind of tests to be placed?

@cfmitrah thoughts on the ACF question?

While we try and avoid calling java in our tests cases, sometimes we need to.

You can do the following

Hi again. Thanks for the test code snippet. I see you can call java code that way… What I was looking for is a way to post form data to Lucee and make sure it gets passed on to the app in the correct encoding. Or something equivalent of testing that. I don’t think there is a quick solution to that.

As for ACF, I noticed that I can call ACF functions from ColdFusion Fiddle ( which appears here and there in ACF’s documentation.
So I checked the behavior of the URLDecode() funciton over there.

I found that ACF doesn’t have issues decoding partially encoded multi-byte sequences, which is good. And it is also just as permissive for erroneous input as Lucee is. It won’t raise exceptions on ill-formed input such as “%#” or just “%”.

So fixing the partially encoded string problem, while maintaining the permissiveness seems to fix an existing compatibility gap without the risk of breaking someone’s (ACF compatible) code.

I will post a PR as follows:

  • For the URLDecoder class, replace it with a hand written version.
  • For the URLDecode() function, maintain the current code structure that first calls the common library, but when that causes an error, give it a second try with URLDecoder(). I will replace the current call to to a call to
1 Like

sounds great, please file the patch against 6.0

thanks for the patch, as per the other path, I have scheduled this for 6.1

1 Like