Query with cachedwithin breaks valuelist

nonfatmedia · April 15, 2024, 10:52pm

I have a simple query that returns 2 rows with 2 columns that uses cachedWithin=“.250”
The 1st row value is 2.
The second row value is 1.

I use valuelist(query.col) to obtain a coma separated list.

The results should be “2,1”. but with lucee 6+ it returns “2,1,” (6 commas after 2,1)

If I remove the cachedWithin=“.250” I get the expected result of “2,1”

also, the docs state that valuelist() is deprecated and we should use [QueryColumnData()] (QueryColumnData() :: Lucee Documentation). QueryColumnData() returns an array. I don’t see how this is a replacement for valuelist().

see screenshot: https://breakdownservices-dev.s3.amazonaws.com/temp/support/13719.png

also: If I attempt to use structappend() with the results of the valuelist(). I get java.lang.ArrayIndexOutOfBoundsException Index 66 out of bounds for length 66

lucee.runtime.exp.NativeException: Index 66 out of bounds for length 66
at lucee.runtime.type.QueryColumnImpl.get(QueryColumnImpl.java:306)
at lucee.runtime.functions.query.ValueList.call(ValueList.java:56)
at lucee.runtime.functions.query.ValueList.call(ValueList.java:48)
at search.index_cfm$cf.call(/search/index.cfm:74)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:1059)
at lucee.runtime.PageContextImpl._doInclude(PageContextImpl.java:951)
at lucee.runtime.listener.ModernAppListener._onRequest(ModernAppListener.java:220)
at lucee.runtime.listener.ModernAppListener.onRequest(ModernAppListener.java:107)
at lucee.runtime.PageContextImpl.execute(PageContextImpl.java:2715)
at lucee.runtime.PageContextImpl._execute(PageContextImpl.java:2701)
at lucee.runtime.PageContextImpl.executeCFML(PageContextImpl.java:2672)
at lucee.runtime.engine.Request.exe(Request.java:45)
at lucee.runtime.engine.CFMLEngineImpl._service(CFMLEngineImpl.java:1259)
at lucee.runtime.engine.CFMLEngineImpl.serviceCFML(CFMLEngineImpl.java:1205)
at lucee.loader.engine.CFMLEngineWrapper.serviceCFML(CFMLEngineWrapper.java:97)
at lucee.loader.servlet.CFMLServlet.service(CFMLServlet.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:596)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:433)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 66 out of bounds for length 66
… 38 more

it also appears that the cached query may get corrupted as once one user/request throws that error every subsequent call to that page throws:
“java.lang.ArrayIndexOutOfBoundsException”
“java.lang.ArrayIndexOutOfBoundsException”
“lucee.runtime.exp.NativeException: java.lang.ArrayIndexOutOfBoundsException
Caused by: java.lang.ArrayIndexOutOfBoundsException”

with no error detail or additional info.

OS: Windows Server 2022 (10.0) 64bit
Java Version: 11.0.18 (Eclipse Adoptium) 64bit
Tomcat Version: Apache Tomcat/9.0.72
Lucee Version: 6.0.1.83

andreas · April 16, 2024, 5:27am

Have you tried QueryColumnData().toList(",")

cfmitrah · April 16, 2024, 6:30am

ValueList is deprecated in lucee. Please should use Andreas suggestion above here.

nonfatmedia · April 16, 2024, 2:57pm

I don’t think its actually valuelist(). I think it has something to do with the cache query itself:

lucee.runtime.exp.NativeException: Index 66 out of bounds for length 66
at lucee.runtime.type.QueryColumnImpl.get(QueryColumnImpl.java:306)

is it possible that since this query was cached by lucee 5.4.3.2 and subsequently called by lucee 6.0.1.83 that there is a caching difference between the 2 versions? We run a cluster of 5.4.3.2 vms all hosting the same site. I updated one of the cluster members to 6.0.1.83. That is when the 6.0.1.83 started throwing errors related to that cached query.

nonfatmedia · April 16, 2024, 5:38pm

I created a test page that calls the same cached query as the one that errors elsewhere and added a try catch and your suggestion to use “QueryColumnData”.

<cftry>
<cfquery datasource="#datasource#" name="vars.qSearchableAgencies" cachedWithin=".250">
<!--- cached for 6 hours --->
SELECT agency_type_id, type_name
FROM dbo.act_agency_type WITH (NOLOCK)
WHERE is_searchable = 1
ORDER BY type_name ASC
</cfquery>

<cfdump var="#vars.qSearchableAgencies#"/>
<cfdump var="#QueryColumnData(vars.qSearchableAgencies,"agency_type_id").tolist(",")#"/>
<cfdump var="#valuelist(vars.qSearchableAgencies.agency_type_id)#"/>
<cfcatch>
<cfdump var="#cfcatch#"/>
</cfcatch>
</cftry>

if I remove cachedWithin=“.250” I get the expected result for both dumps without error. 2,1 & 2,1.

nonfatmedia · April 16, 2024, 5:59pm

also, If I loop over the cache query:<cfloop query="#vars.qSearchableAgencies#"> #vars.qSearchableAgencies.agency_type_id#<br> </cfloop>
it returns
2
1
as expected.

Terry_Whitney · April 16, 2024, 6:00pm

You have a dot in front of the 250, is that by design?

=.250

nonfatmedia · April 16, 2024, 6:20pm

yes. fraction of a day. 6 hours.

Terry_Whitney · April 16, 2024, 6:33pm

Thats not how that tag works.

you are telling a int value to be a fractual value.

This catchedwithin works off the same tag as createTimeSpan,

so your code should be ```
cachedWithin=" #createTimespan(0, 6, 0, 0)#">

nonfatmedia · April 16, 2024, 7:48pm

what is value returned from #createTimespan(0, 6, 0, 0)#. if I use .250 or createTimespan(0, 6, 0, 0) makes no difference. That is interesting though. In past when you were to call createTimespan(0, 6, 0, 0)
it would return a decimal value. Now it returns the the string “createTimespan(0, 6, 0, 0)”. I don’t think that cfquery cares which one you use. and in my case it doesn’t change the outcome.

kenricashe · April 16, 2024, 9:42pm

CFDocs site claims that CreateTimespan() returns numeric (double), but Lucee docs says it returns Timespan object. A cfoutput will indicate 0.25, but a cfdump will indicate the Timespan value represented as a string, but the return value is indeed a Timespan object. Check your code again and make sure it’s enclosed in ##? It’s entirely possible you already have it enclosed properly, but I just wanted to note the differences in how the function is documented and its cfoutput vs cfdump representations.

Terry_Whitney · April 17, 2024, 11:34am

without the complete code, or at least a large section to sample, its hard to recreate your error.

Perhaps post the code in a gist from trycf.com as more people looking at it may be able to solve the issue.

nonfatmedia · April 17, 2024, 2:01pm

The complete code that throws the error is

<cftry>
<cfquery datasource="#datasource#" name="vars.qSearchableAgencies" cachedWithin=".250">
<!--- cached for 6 hours --->
SELECT agency_type_id, type_name
FROM dbo.act_agency_type WITH (NOLOCK)
WHERE is_searchable = 1
ORDER BY type_name ASC
</cfquery>

<cfdump var="#vars.qSearchableAgencies#"/>
<cfdump var="#QueryColumnData(vars.qSearchableAgencies,"agency_type_id").tolist(",")#"/>
<cfdump var="#valuelist(vars.qSearchableAgencies.agency_type_id)#"/>
<cfcatch>
<cfdump var="#cfcatch#"/>
</cfcatch>
</cftry>

One thing I noticed though is that this query will not fail immediately but once it does it fails continuously. This has something to do with the lucee update.

Terry_Whitney · April 17, 2024, 4:05pm

It maybe a bug, but it could be a memory leak.

I would look at the underlying java engine, update that as well as the memory state of your server.

kenricashe · April 17, 2024, 5:57pm

Share the code where you’re using CreateTimespan() inside the cachedWithin argument?

nonfatmedia · April 17, 2024, 6:23pm

same result with CreateTimespan(). it appears that the cached query is corrupt somehow.

<cftry>
<cfquery datasource="#datasource#" name="vars.qSearchableAgencies" cachedWithin="#CreateTimespan(0,6,0,0)#">
<!--- cached for 6 hours --->
SELECT agency_type_id, type_name
FROM dbo.act_agency_type WITH (NOLOCK)
WHERE is_searchable = 1
ORDER BY type_name ASC
</cfquery>

<cfdump var="#vars.qSearchableAgencies#"/>
<cfdump var="#QueryColumnData(vars.qSearchableAgencies,"agency_type_id").tolist(",")#"/>
<cfdump var="#valuelist(vars.qSearchableAgencies.agency_type_id)#"/>
<cfcatch>
<cfdump var="#cfcatch#"/>
</cfcatch>
</cftry>

nonfatmedia · April 17, 2024, 6:31pm

What is interesting as well. is that we have over a thousand valuelist() and cachedwithin queries in our code base and this is the only one that has so far failed.

TonyMonast · April 18, 2024, 6:52pm

Hi

What happens if you add in a SQL query a fake column like

SELECT agency_type_id, type_name, 1 AS DummyTest

Run the test twice and see if the result is the same for ValueList? And do both tests with cachedWithin=“#CreateTimespan(0,6,0,0)#” and not your .250 value

If the problem persists, change again the SELECT to make sure it’s a new query and try a different cache lifetime, like 15 minutes.