Lucee 6.0.x.x Canonicalize Exception

Conrad-T-Pino · March 28, 2024, 3:34pm

I run all form data through Lucee Canonicalize function using a component static function:

public static string function canonical(required string data) {
	return canonicalize(data, false, false);
}

The attached exception is triggered by password ending with percent sign: Zappar1234%
I was expecting default throwOnError=false argument to prevent all exceptions.
I’m unclear on Canonicalize() required semantics. What should happen here?

OS: Linux 5.10.0-28-amd64 SMP Debian 5.10.209-2 (2024-01-31) x86_64 GNU/Linux
Java Version: OpenJDK 64-Bit Server VM (build 11.0.22+7-post-Debian-1deb11u1, mixed mode, sharing)
Tomcat Version: Tomcat 9.0.43-2~deb11u9
Lucee Version: 6.0.0.585 and 6.0.1.83

Message: URLDecoder: Incomplete trailing escape (%) pattern
StackTrace: lucee.runtime.exp.NativeException: URLDecoder: Incomplete trailing escape (%) pattern
at java.base/java.net.URLDecoder.decode(URLDecoder.java:225)
at java.base/java.net.URLDecoder.decode(URLDecoder.java:142)
at org.lucee.extension.esapi.functions.Canonicalize.call(Canonicalize.java:48)
at org.lucee.extension.esapi.functions.Canonicalize.invoke(Canonicalize.java:70)
at lucee.runtime.functions.FunctionHandlerPool.invoke(FunctionHandlerPool.java:40)
at formutil_cfc$cf$2.udfCall2(/FormUtil.cfc:414) at formutil_cfc$cf$2.udfCall(/FormUtil.cfc)
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:356) at lucee.runtime.type.UDFImpl.call(UDFImpl.java:223)
at lucee.runtime.StaticScope._call(StaticScope.java:342)
at lucee.runtime.StaticScope.call(StaticScope.java:298)
at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:787)
at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1777)
at formutil_cfc$cf$2.udfCall3(/FormUtil.cfc:578)
at formutil_cfc$cf$2.udfCall(/FormUtil.cfc)
at lucee.runtime.type.UDFImpl.implementation(UDFImpl.java:112)
at lucee.runtime.type.UDFImpl._call(UDFImpl.java:356)
at lucee.runtime.type.UDFImpl.call(UDFImpl.java:223)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:701)
at lucee.runtime.ComponentImpl._call(ComponentImpl.java:589)
at lucee.runtime.ComponentImpl.call(ComponentImpl.java:1993)
at lucee.runtime.util.VariableUtilImpl.callFunctionWithoutNamedValues(VariableUtilImpl.java:787)
at lucee.runtime.PageContextImpl.getFunction(PageContextImpl.java:1777)
at webreg.delekhopen24.action.login_cfm$cf$92.call(/webreg/DelekHopeN24/action/login.cfm:15)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1058)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:950)
at lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:221)
at lucee.runtime.listener.MixedAppListener.onRequest(MixedAppListener.java:44)
at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2494)
at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2480)
at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2451)
at lucee.runtime.engine.Request.exe(Request.java:45)
at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1219)
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1165)
at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)
at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:667)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888)
at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:1961)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: URLDecoder: Incomplete trailing escape (%) pattern ... 59 more

Lucee_Canonicalize_Exception.mhtml (15.1 KB)

andreas · March 28, 2024, 4:25pm

I think this may be related
https://luceeserver.atlassian.net/browse/LDEV-4743

Conrad-T-Pino · March 28, 2024, 4:32pm

This is old:

Post 10675 Jul 2022 Canonicalize() - Illegal hex characters in escape (%) pattern
Post 13181 Nov 2023 Canonicalize error with latest ESAPI 2.2.4.15

Conrad-T-Pino · March 29, 2024, 2:50am

I git cloned Lucee repository to review Canonicalize implementation and conclude:

While Lucee source has test compoment: Lucee/test/functions/Canonicalize.cfc
Lucee doesn’t implement Canonicalize function directly;
somehow runtime maps execution into ESAPI extension.
OWASP ESAPI JavaDoc have:
3.1. Encoder (ESAPI 2.2.3.1 API)
3.2. DefaultEncoder (ESAPI 2.2.3.1 API)
Which confirm Lucee documentation Canonicalize() fourth throwOnError argument is ignored when either 2nd or 3rd argument is false.
Passing both 2nd and 3rd arguments as true makes no difference; exception still thrown.
Would it be easy or hard to make 4th argument effective in all cases?
Does the 4th argument play any role at all in the implementation?
Found it: source/java/src/org/lucee/extension/esapi/functions/Canonicalize.java

Inspecting class org.owasp.esapi.reference.DefaultEncoder with CreateObject shows three canonicalize methods with at most three arguments:

Conrad-T-Pino · March 29, 2024, 4:57am

component {
	processingdirective preserveCase=true;

	// java.net.URLDecoder.decode(String, Charset) throws
	// IllegalArgumentException if data has malformed % encoding
	// For example password: Zappar1234% or Za%ppar1234
	public static string function canonical(required string data) {
		return Canonicalize(data, false, false);
	}

	// Lucee ESAPI extension "Canonicalize" function replacement
	static {
		private classESAPI = CreateObject("java", "org.owasp.esapi.ESAPI");
	}

	public static string function canonical(required string data) {
		var Encoder = static.classESAPI.encoder();
		return Encoder.canonicalize(data, false, false);
	}
}

Conrad-T-Pino · March 29, 2024, 4:23pm

There is no doubt LDEV-4743 and this issue have same underlying implementation issue:

source/java/src/org/lucee/extension/esapi/functions/Canonicalize.java Line 48
java.net.URLDecoder.decode is unforgiving of malformed percent-encoding
and multiple level percent-encoding is ignored.

In my case, percent is wanted as password special character and in business or financial text fields. My data sources are URI and FORM scopes; both of which are already URL decoded.

IMO it’s debatable if Canonicalize() should URL decode at all but assuming it should, the implementation should handle multiple level, if enabled, and support leaving malformed percent-encoding as is.

Is there any record of how Canonicalize() was designed or implemented?

Conrad-T-Pino · March 30, 2024, 5:25am

GitHub source code:

source/java/src/org/lucee/extension/esapi/functions/Canonicalize.java

has three commits:

Dec 21, 2017 Commit 2b978af8c31852beba7ef982c55ea41686d38ad2
Feb 6, 2022 Commit bc27485337adf51a67c03eeaedaaa903f2b4c951
Sep 8, 2023 Commit c4e40a4392dcb577131a56fac90217b7976fdee9

The unforgiving java.net.URLDecoder.decode invocation was added Feb 6, 2022.

The Sep 8, 2023 commit added java.io.UnsupportedEncodingException try catch that ignores current java.lang.IllegalArgumentException issue which as an unchecked RuntimeException percolates up the call stack unwrapped (IMO another bug).

IMO we should consider removing java.net.URLDecoder.decode or replacing with better implementation. I’d like to consult with @michaeloffner as well since he is original author and latest contributer. Signaling @micstriit for consultation.

kenricashe · March 30, 2024, 7:26pm

Count me as one of many who’ll benefit from a fix for this.