SerializeJSON with lowercase keys and override JSONConverter.java

Hi. I am very new to Lucee and loving it! Thanks!

I am migrating a CF10 legacy app and all is going well so far.
This legacy app basically provides JSON packets for a client side JS app.
As such the client app is expecting all json keys to be lower case. The CF10 app uses Ben Nadel’s JsonSerializer to accomplish that (as well as covering the other ACF serialization issues). That JsonSerialiser has been working well for years.

Now, under Lucee it is really, really slow compared to Lucee’s SerializeJSON. As a separate exercise I need to work out why (and have no clue yet).

The question here, though, is given SerializeJSON fixes all the other ACF issues, the only remaining one being the need to generate lowercase keys, I was thinking it would be great to leverage SerializeJSON and override the JSONConverter.java file, simply to set lower case keys instead of preserving or upper, which are the only two choices at present. (As an aside, it would be great to have the 3 choices: preserve, upper or lower)

I know I can use preserve case (and am doing that) but there are simply too many places to go through this legacy app (which has a future lifespan of about 6 months) and change the struct assignments to ensure lowercase there. I really need to do that as part of the serialization step.

So, my question is, what would be the process of using a modified version of JSONConverter.java class. I have some limited java skills but am not sure how to accomplish this override in Lucee.

Thanks again,
Murray

1 Like

I’d expect you’d have to build your own fork of Lucee. If there’s an easier alternative to that, I don’t know so I’ll leave an actual answer to others with more knowledge.

Short of that, is it feasible in your case to do a regex search/replace or create a script to modify all your .cf* files?

Thanks @LionelHolt . yes, possibly. I was trying to avoid that and also would prefer to shift the responsibility to the relevant place (the serialiser) since struct key case is irrelevant to me apart from when turning it into json.

Another option I am playing with is to run a recursive function in my custom serialiser function that converts the nested structs / arrays into a copy with lowercase keys before serialising that copy. I am working on it just out of interest to see if the overhead is worth it. The Lucee serializer is fast and the minor adjustments that class to convert keys to lowercase seems to be the simplest way to go - especially if I can somehow “override” that class to do what I need without needing to fork Lucee itself.

Much appreciated,
Murray

1 Like

To be clear, ACF probably hasn’t had the JSON serialization issues you’re thinking of in many years. Most of those were fixed in 2016, 2018, an 2021. Only people waaay back on ACF10 would need to worry about that :wink:

This is really your “quick fix”

This is the “real” answer IMO. And honestly, I think you’re overstating the work. It’s not too hard to change

var myStr = {
  foo : 'bar',
  baz : 'bum'
}

to

var myStr = {
  'foo' : 'bar',
  'baz' : 'bum'
}

or

myStr.brad = 'wood';

to

myStr[ 'brad' ]= 'wood';

Sure, it will be a little annoying, but you and your co workers can probably knock it all out in an afternoon.

Don’t even believe those statements. As a CF contractor, I can’t recall how many companies I’ve worked with who were “6 months” from retiring CF code and here it is years and years later grinding on in production. :slight_smile:

4 Likes

What @bdw429s said is of course the best choice. Don’t know your code and how much files you have or need to change.

Also, I’d not change the core of Lucee. As soon as you get an urgent update, you’d need to at least rebuild it and deploy it with your changed core code.

My alternative would be (like you already said) create a helper component that has a function that converts the keynames (just like you said recursively). I didn’t test this extensively, but here is something that worked quickly and I’d try that as a practical approach. However, that would create more overhead. You should really consider what @bdw429s said. Here is the other option:

MyJsonConverter.cfc

component displayname="MyJsonConverter.cfc" output="false" {
	
    public struct function init(){
        
        return this;

    }


    private struct function lowerStructKeyNames ( any dataobject required ){

         local.result=[:];

        arguments.dataobject.each( function( key, value ) {

                if( isStruct( arguments.value )){

                    result.append( { "#lcase(arguments.key)#": lowerStructKeyNames ( arguments.value ) } );

                }else{

                    result.append( { "#lcase(arguments.key)#": arguments.value } );
                
                }

            }

        );
        
        return local.result;

    }


    public string function serializeJsonWithLowerKeyNames ( any dataobject required ){

        return serializeJSON( lowerStructKeyNames( arguments.dataobject) );

    }


}


Then in your code you would need to add that component in global manner e.g. myJsonConverter= new MyJsonConverter(); and replace your serializeJSON() function in your code just like @LionelHolt already mentioned call it with MyJsonConverter.serializeJsonWithLowerKeyNames( ) instead:

example.cfm:

<cfscript>
myJsonConverter= new MyJsonConverter();
dump( serializeJson( getApplicationSettings(false) ) );
dump( MyJsonConverter.serializeJsonWithLowerKeyNames( getApplicationSettings(false) ));
</cfscript>
1 Like

Thanks to all for your quick responses. :slight_smile:
I will digest, test, and report back.
Cheers,
Murray

1 Like

I’ve just seen that my component will return some struct as strings, and that wouldn’t work. But it may serve as a starting point.

it was too early in the morning here when I wrote that code. I was calling the wrong function as the recursion function. Just updated it.

Thanks @andreas - that’s helpful. It’s getting late here so I will return to this in the morning. Much appreciated.

1 Like

So, reporting back as promised…
In the end I went with a modified version of the code @andreas suggested. See below. Thank you! :slight_smile:

  myJsonConverter = new MyJsonConverter();
  j1 = MyJsonConverter.serializeJsonWithLowerKeyNames( myNestedStruct );
  j2 = MyJsonConverter.serializeJsonWithLowerKeyNames( myArrayOfNestedStructs );

In the CF10 legacy code, I am using Taffy and the serialization is done in just one place, so having this there simplifies the process as I can catch all requests for JSON in the one place. That was preferable to manually changing all the structs at origin because there were many and there was a good chance of missing some, given the way the data objects were being constructed (and it is just me - no team :frowning: )

In terms of overhead, unless the object being serialized is very large, the speed is acceptable:

Average test results comparing the native Lucee SerializeJSON and the convert-to-lowercase function:
17 small records: 985 bytes of json:
native: 85 ms
lowercased: 102 ms

13 large, nested records, 34 Kb of json
native: 5268 ms
lowercased: 5340 ms

3,450 large nested records, 1.4 Mb of json Yes - big!
native: 6630 ms
lowercased: 13320 ms

I also made a language proposal here: Topic 10636

/**
  CFC to call the Lucee serializeJSON() after converting all the keys
  in the passed in data object to lowercase.

  eg:
  myJsonConverter = new MyJsonConverter();
  j1 = MyJsonConverter.serializeJsonWithLowerKeyNames( myNestedStruct );
  j2 = MyJsonConverter.serializeJsonWithLowerKeyNames( myArrayOfNestedStructs );

  See: https://dev.lucee.org/t/serializejson-with-lowercase-keys-and-override-jsonconverter-java/10619
  for the rationale and
  credit to @andreas for the initial code in that post. :-)
*/
component displayname="MyJsonConverter.cfc" output="false" {

    public struct function init(){

        return this;

    }

    /**
      PUBLIC
      Uses the Lucee native serializeJSON() to serialize the object to JSON,
      AFTER converting all keys to lowercase.
    */
    public string function serializeJsonWithLowerKeyNames ( any dataobject required ){
        if (isArray( dataobject )) {
          // We might start with an array at the top level
          return "[#serializeJSON( lowerStructKeyNames( arguments.dataobject[1]) )#]";
        } else {
          // Or a struct, or simple value
          return serializeJSON( lowerStructKeyNames( arguments.dataobject ) );
        }
    }

    /**
      PRIVATE
      If the dataobject is a struct or array,
      for all keys in that object convert them to lower case
      and return a new object with those lower case keys.
      If the dataobject is a simple value, just return it unchanged.
    */
    private any function lowerStructKeyNames ( any dataobject required ){
        if (isSimpleValue(dataobject)) {

          // Just return the simple value unchanged
          local.result = dataobject;

        } else {
          local.result=[:];

          arguments.dataobject.each( function( key, value ) {
              if( isStruct( arguments.value )){

                  // Structs recurse ...
                  result.append( { "#lcase(arguments.key)#": lowerStructKeyNames ( arguments.value ) } );

              } else if (isArray( arguments.value )){
                  // Array, iterate and recurse ...
                  var arr = [];
                  arguments.value.each( function (item) {
                    arr.append( lowerStructKeyNames ( item ) );
                  });

                  result.append( { "#lcase(arguments.key)#": #arr# } );
              } else {
                  // Everything else, just transform
                  result.append( { "#lcase(arguments.key)#": arguments.value } );
              }
          });
        }

        return local.result;
    }
}

1 Like

@flowt-au Many thanks for posting back your findings, sharing your code and the timing responses! Great help for others who might have similar issues.

I LOVE Taffy!
The dashboard (that works) in 3.6 is a godsend for testing!

@Gavin_Baumanis Indeed! I was using a modified version of v1 in the old CF10 app and now v3 seems to work “out of the box” for my needs. :slight_smile:

I can’t remember off the top of my head…
But there were some insignificant changes needed to use 3.6 over 3.

Nonetheless - I am a fan.

@Gavin_Baumanis Yes, v3.6. All works well. I created a custom serializer so all the toLowerCase process was taken care of there and I could leave the rest of the CFCs that generated the arrays of structs independant of the need to worry about key case. I also abstracted those “resource” CFCs so I only have 4 end point taffy_uri patterns that route to the “real” resources that do the work. That way those “real resource” CFCs also know nothing about Taffy - they just return structs or arrays of structs back to Taffy who serializes them for the client app. Works well. :slight_smile:

Update: I am now not so clueless. Docker noob. I am on a Win 10 box using VSCode. To avoid complete overload I began my journey running Docker Desktop without the VSCode remote WSL approach. Realising this was what was slowing the whole app down, including the serialization, I learnt how to use the VSCode Windows remote WSL Ubuntu approach to make the containers inside Ubuntu and voila! instant dramatic speed improvement. Phew!

2 Likes

Would be super interesting to see your journey to make that work. In case you make a blog post about that specific setup with Lucee/Docker/WSLUbuntu/VScode, please let me know. I’d love to read it.

1 Like

@andreas Yes, I thought I might. As a newby to Lucee + MySql (MariaDB) + nginx AND Docker AND WSL AND Ubuntu I found the whole exercise having a very steep learning curve! I am still in the weeds making everything work. Mostly done, just stuck on a db corruption issue now. :frowning:

Here it is: https://dev.lucee.org/t/tutorial-lucee-docker-and-vscode-in-2022/10857

Cheers,
Murray

1 Like

Not specifically Lucee, but performance tweak none the less
on debian / ubuntu,
if your kernel is 4.9 or newer.
Check by using
uname -r

if so add the following to the end of
/etc/sysctl.conf
ie
vi /etc/sysctl.conf

lines to add
net.core.default_qdisc=fq
net.ipv4.tcp_congestion_control=bbr

save and reload your network stack
sysctl -p

1 Like