Lucee 5 Maven Builds Prototype

A lot of work has been going on behind the scenes re-architecting the Lucee 5 code base to build out of Maven central. That effort is beginning to bear fruit :apple:

For those interested in poking around, @micstriit has given me permission to post up some of the prototypes for working with the new Maven repo. If you prefer your distributions already pre-baked into a more standard distribution, you won’t have long to wait.

The code attached gives access to all kinds of data from Maven. The implementation is inside MavenRepo.cfc, and the index.cfm provides examples to do the following:

  • get a list of all available builds (snapshots, releases or all) with detailed info (build date, jar and pom location)
  • get the latest snapshot/release with detailed info (build date, jar and pom location)
  • get detailed info for a specific version (build date, jar and pom location)
  • read all dependencies from a POM
  • get (downloaded and generated if necessary) a zip containing all jars for a specific version
  • get (downloaded and extracted from jar if necessary) lucee core file (.lco)

The Lucee 5 update provider will use similar code, so in the end there will also be various endpoints where you can go to get specific distributions; for example, the complete jars zip.

Basic index.cfm for pulling various incantations from the Maven repo.

<cfsetting requesttimeout="100000">
<cfscript>
mr=new MavenRepo();

/**************************************
***************** ALL *****************
get all available (snapshot|release) versions
**************************************/
/*
// get all available releases
releases=mr.getAvailableVersions("releases"); // simple (fast)
dump(var:releases,label:"releases simple");
releases=mr.getAvailableVersions("releases",true); // extended (slow)
dump(var:releases,label:"releases extended");

// get all available snapshots
snapshots=mr.getAvailableVersions("snapshots"); // simple (fast)
dump(var:snapshots,label:"snapsots simple");
snapshots=mr.getAvailableVersions("snapshots",true); // extended (slow)
dump(var:snapshots,label:"snapsots extended");
*/

/**************************************
*************** LATEST ****************
get info for the latest snapshot/release
**************************************/
/*
// get latest snapshots
snapshot=mr.getAvailableVersions("snapshots",false,true);
dump(var:snapshot,label:"latest snapsot simple");
snapshot=mr.getAvailableVersions("snapshots",true,true);
dump(var:snapshot,label:"latest snapsot extended");

// get latest releases
snapshot=mr.getAvailableVersions("releases",false,true);
dump(var:snapshot,label:"latest release simple");
snapshot=mr.getAvailableVersions("releases",true,true);
dump(var:snapshot,label:"latest release extended");
*/

/**************************************
*************** Dependencies *************
get depencies for a POM
**************************************/
/*
snapshot=mr.getAvailableVersions("releases",true,true); // get the latest release
dep=mr.readDependenciesFromPOM(snapshot.pomSrc); // get dependcies
dump(dep);
*/

/**************************************
*************** Specific ****************
**************************************/
info=mr.getInfo("5.0.0.60-BETA");
dump(var:info,label:"Info for 5.0.0.60-BETA");

/**************************************
*********** create jars zip ***********
**************************************/
dump(mr.getJars("5.0.0.62-BETA"));

/**************************************
*********** get core  ***********
**************************************/
dump(mr.getCore("5.0.0.62-SNAPSHOT"));


//dump(readDependenciesFromPOM("https://oss.sonatype.org/content/repositories/releases/org/lucee/lucee/5.0.0.62-BETA/lucee-5.0.0.62-BETA.pom"));

</cfscript>

And the prototype component MavenRepo.cfc that’s doing all the heavy lifting.

component {
	variables.listPattern=	"https://oss.sonatype.org/service/local/lucene/search?g={group}&a={artifact}";
	//defaultRepo="https://oss.sonatype.org/content/repositories/releases";
	variables.defaultRepo="http://central.maven.org/maven2";
	variables.group="org.lucee";
	variables.artifact="lucee";

	// this are version that should not be used in any case
	variables.ignoreVersions=["5.0.0.22","5.0.0.travis-74435522-SNAPSHOT"];

	public function readDependenciesFromPOM(string pom, boolean extended=false){		

		local.content=fileRead(pom);
		local.xml=xmlParse(content);

		// custom repositories
		local.repos=[defaultRepo];
		local.xmlRepos=xml.xmlRoot.repositories.xmlChildren;
		loop array=xmlRepos item="local.xmlRepo" {
			arrayAppend(repos,xmlRepo.url.xmlText);
		}
		// dependencies
		local.dependencies=xml.xmlRoot.dependencies.xmlChildren;
		local.arr=[];
		if(isNull(application.repoMatch))application.repoMatch={};
		loop array=dependencies item="local.dependency" {
			local.g=dependency.groupId.xmlText;
			local.a=dependency.artifactId.xmlText;
			local.v=dependency.version.xmlText;
			local.id=g&":"&a&":"&v;
			
			if(	"org.apache.felix"==g || 
				"junit"==g || 
				"javax.servlet"==g || 
				"javax.servlet.jsp"==g || 
				"javax.el"==g ||
				"org.apache.ant"==g) continue;
			
			// find the right repo
			//dump(id);
			// TODO store physically
			if(!isNull(application.repoMatch[id])) {
				local.detail=application.repoMatch[id];
				//dump("cached");
			}
			else {
				local.start=getTickCount();
				local.detail={};
				loop array=repos item="local.repo" {
					try{local.tmp=getDetail(repo,g,a,v,true);}catch(e){continue;};
					if(!isNull(tmp.jar.src)) {
						detail.jar=tmp.jar.src;
						if(!isNull(tmp.pom.src))detail.pom=tmp.pom.src;

						dump(detail);
						application.repoMatch[id]=detail;
						break;
					}
				}
				local.detail.groupId=g;
				local.detail.artifactId=a;
				local.detail.version=v;
				systemOutput(id&":"&(getTickCount()-start),true,true);
				
			}
			
			arrayAppend(arr,detail);			
			
		}
		return arr;
	}

	/**
	* return information about a specific version
	* @version version to get info for
	*/
	public struct function getInfo(required string version){
		local.qry= getAvailableVersions(type:'all', extended:true, onlyLatest:false,specificVersion:version);
		if(qry.recordcount==0) throw "no info found for version ["&version&"]";
		return QueryRowData(qry,1);
	}

	/**
	* return information about available versions
	* @type one of the following (snapshots,releases or all)
	* @extended when true also return the location of the jar and pom file, but this is slower (has to make addional http calls)
	* @onlyLatest only return the latest version
	*/
	public query function getAvailableVersions(string type='all', boolean extended=false, boolean onlyLatest=false){

		// validate input
		if(type!='all' && type!='snapshots' && type!='releases')
			throw "provided type [#type#] is invalid, valid types are [all,snapshots,releases]";

		// create the list URL
		local.listURL=convertPattern(listPattern,group,artifact);
		
		// get data
		http url=listURL result="local.res" {
			httpparam type="header" name="accept" value="application/json";
		}
		local.raw=deSerializeJson(res.fileContent);
		
		// repo urls
		if(extended) local.repos=getRepositories(raw.repoDetails);

		// create the list
		local.qry=queryNew("groupId,artifactId,version,type"&(extended?",pomSrc,pomDate,jarSrc,jarDate":""));
		loop array=raw.data item="local.entry" {
			if(isNull(entry.artifactHits[1])) continue;
			local.ah=entry.artifactHits[1];

			// ignore list
			if(arrayContains(variables.ignoreVersions,entry.version)) continue;
			// check type
			if(type!="all" && type!=ah.repositoryId) continue;
			// latest
			if(onlyLatest && entry.version!=entry.latestSnapshot && entry.version!=entry.latestRelease) continue;
			// specific
			if(!isNull(specificVersion) && specificVersion!=entry.version) continue;

			local.row=qry.addRow();

			if(extended)local.sources=getDetail(repos[ah.repositoryId],entry.groupId,entry.artifactId,entry.version);

			//dump(sources);
			qry.setCell("groupId",entry.groupId,row);
			qry.setCell("artifactId",entry.artifactId,row);
			qry.setCell("version",entry.version,row);
			qry.setCell("type",ah.repositoryId,row);
			
			if(extended) {
				if(!isNull(sources.pom.src))qry.setCell("pomSrc",sources.pom.src,row);
				if(!isNull(sources.pom.date))qry.setCell("pomDate",sources.pom.date,row);
				if(!isNull(sources.jar.src))qry.setCell("jarSrc",sources.jar.src,row);
				if(!isNull(sources.jar.date))qry.setCell("jarDate",sources.jar.date,row);
			}
			//else qry.setCell("artifacts",ah.artifactLinks,row);
		}
		if(extended)querySort(qry,"jarDate","asc");
		else querySort(qry,"version","asc");
		return qry;
	}

	function getRepositories(required array repoDetails) {
		local.repos={};
		loop array=repoDetails item="local.entry" {
			http url=entry.repositoryURL result="local.res" {
				httpparam type="header" name="accept" value="application/json";
			}
			local._raw=deSerializeJson(res.fileContent);
			repos[entry.repositoryId]=_raw.data.contentResourceURI;
		}
		return repos;
	}

	public struct function getDetail(required string repoURL,
			required string groupId,required string artifactId,required string version, 
			boolean simply=false){
		local.base=repoURL&"/"&replace(groupId,'.','/',"all")&"/"&replace(artifactId,'.','/',"all")&"/"&version;

		local.sources={};
		if(!simply){
			http url=base&"/maven-metadata.xml" result="local.content" {
				httpparam type="header" name="accept" value="application/json";
			}
		}
		// read the files names from xml
		if(!simply && content.status_code==200) {
			local.xml=xmlParse(content.fileContent);
			loop array=xml.XMLRoot.versioning.snapshotVersions.xmlChildren item="node" {
				local.date=toDate(node.updated.xmlText,"GMT");
				local.src=base&"/"&artifactId&"-"&node.value.xmlText&"."&node.extension.xmlText;
				sources[node.extension.xmlText]={date:date,src:src};
			}
		}
		// if there is no meta file simply assume
		else {
			// date jar
			try{
				http method="head" url=base&"/"&artifactId&"-"&version&".jar" result="local.t";
				if(t.status_code==200) {
					local.sources.jar.src=base&"/"&artifactId&"-"&version&".jar";
					local.sources.jar.date=parseDateTime(t.responseheader['Last-Modified']);
				}
			}
			catch(e){}
			// date pom
			try{
				http method="head" url=base&"/"&artifactId&"-"&version&".pom" result="local.t";
				if(t.status_code==200) {
					local.sources.pom.src=base&"/"&artifactId&"-"&version&".pom";
					local.sources.pom.date=parseDateTime(t.responseheader['Last-Modified']);
				}
			}
			catch(e){}
		}
		return sources;
	}


	private function convertPattern(required string pattern,string group,string artifact){
		local.rtn=replace(pattern,'{group}',group,'all');
		local.rtn=replace(rtn,'{artifact}',artifact,'all');
		return rtn;
	}

	private function toDate(string str,string timeZone){
		return createDateTime(
				mid(str,1,4),
				mid(str,5,2),
				mid(str,7,2),
				mid(str,9,2),
				mid(str,11,2),
				mid(str,13,2),0,timezone
		);

	}

	/**
	* returns local location for the core of a specific version (get downloaded if necessary)
	* @version version to get jars for, can also be 
	*/
	public string function getCore(required string version) {
		local.info=getInfo(version); // get info for defined version
		// create artifacts directory
		local.dir=getDirectoryFromPath(getCurrenttemplatePath())&"artifacts/";
		if(!directoryExists(dir))directoryCreate(dir);
		
		local.temp=dir&"tmp#rand()#.jar"; // temp file to store the downloaded jar
		local.trg=dir&info.version&".lco" // target lco file
		if(!fileExists(trg)) {
			try {
				fileCopy(info.jarSrc,temp); // get it local
				fileCopy("zip://"&temp&"!core/core.lco",trg); // now extract 
			}
			finally {
				fileDelete(temp);
			}
		}
		return trg;
	}
	

	/**
	* returns local location for the core of a specific version (get downloaded if necessary)
	* @version version to get jars for, can also be 
	*/
	public string function getJars(required string version) {
		local.info=getInfo(version); // get info for defined version
		local.dir=getDirectoryFromPath(getCurrenttemplatePath())&"artifacts/";
		if(!directoryExists(dir))directoryCreate(dir);
		local.zip=dir&info.version&".zip";
		if(!fileExists(zip)) {
			local.dependencies=readDependenciesFromPOM(info.pomSrc); // get dependcies

			// let's zip it
			zip action="zip" file=zip overwrite=true {
				// all dependcies
				loop array=dependencies item="local.dep" {
					zipparam source=dep.jar;
				}
			}
		}
		return zip;
	}


}

The component will be integrated into the update provider, so you will be able to do things like:

http://stable.lucee.org/rest/update/provider/dependencies/5.0.0.62-BETA

The script gets the jars from maven when necessary and creates the zip on the fly when necessary; @micstriit promises more to come!

1 Like

you can also find everything build automatically by lucee here:
http://preview.lucee.org/download/