formatBaseN returns different results in CF and Lucee

I’m trying to step through the polyline algorithm that Google uses in their mapping and test my implementation in Lucee.

On step 3, the task is to convert the decimal value to binary, calculating negative values using a two’s complement. In CF, that is exactly what formatBaseN() does. In Lucee however, it represents it differently.

Lucee
formatBaseN( -17998321, 2 ) => -1000100101010000111110001

CF
formatBaseN(-17998321, 2) => 11111110111011010101111000001111

Is there a handy function to get the same result as I did from CF?

Edit: I ended up tapping into Mapbox’s Java SDK to achieve the encoded polylines. Still interested in knowing why this difference exists though.

@daamsie Yes, using negative numbers in formatBaseN() returns different results in Lucee and ACF. Can you please file incompatibility for this issue here https://luceeserver.atlassian.net/

looks like adobe only accepts int32 here, while lucee accepts int64s.

we can simulate adobe’s result by masking the int64 to an int32 (not saying it’s desirable this way, but it is illustrative)

low32_mask = 2^32-1; // 0xFFFFFFFF
v = -17998321; // signed representation is 0xFFFFFFFFFEED5E0F

// internally this deals with java ints so we lose the top bit of the mask to the sign
// pretending we saved that bit we get 0x00000000FEED5E0F
// which is a positive number and so stringifies as desired
v_masked = bitand(v, low32_mask);

s = formatBaseN(v_masked, 2)
writedump(s)

There is no way from cf to bitAndUnsigned or similar, so we lose the topmost bit.

formatBaseN is a thin wrapper around Long.toString(long) and the “leading minus stringification” is expected behavior when passing a negative long, of course it may not be desireable CF behavior.

cheers bro

The way Google says to do it in their polyline encoder algorithm is as follows:

… a negative value must be calculated using its two’s complement by inverting the binary value and adding one to the result:

00000001 00010010 10100001 11110001
11111110 11101101 01011110 00001110
11111110 11101101 01011110 00001111

Which mirrors the end result from ACF’s FormatBaseN(int, 2)

public class MyClass {
    public static void main(String args[]) {
        long value = 0xFFFFFFFF_FEED5E0FL; // -17998321, 2's complement
        long lucee = value;
        long acf = value & 0x00000000_FFFFFFFFL;
        
        System.out.println((long)lucee);
        System.out.println((int)acf);
        
        System.out.println("acf: " + Long.toString(acf, 2));
        System.out.println("lucee: " + Long.toString(lucee, 2));
    }
}

adobe looks like it’s discarding the high bits of a java long, just dealing with int32s, arguably with good reason, since the output for base 2 with their way is way more expected