Building lucee documentation on windows and performance tuning

Zackster · March 23, 2018, 4:49am

I just finally got around to porting the docs build.sh and serve.sh over to batch files for windows

I hit the same error which the travis-ci is also hitting, it’s because the download folder doesn’t have a number for the sort order. doesn’t look like anything has changed recently, except maybe it’s using the 5.2.6.60 release?

https://travis-ci.org/lucee/lucee-docs/builds/357232550?utm_source=github_status&utm_medium=notification

[expression] error occurred while building the docs. Message: [invalid call of the function setSortOrder (/home/travis/build/lucee/lucee-docs/api/data/Page.cfc), first Argument (sortOrder) is of invalid type, can't cast String [] to a value of type [numeric]]. Detail: [].

Zackster · March 23, 2018, 6:07am

anyone familiar with the docs cfml code?

I think it’s doing way more recursive work than it needs to?

i don’t think page.getChildren() should be passed in to _calculateNextAndPreviousPageLinks again

github.com

lucee/lucee-docs/blob/master/api/data/DocTree.cfc#L236


      
          		lock name="docsBuildTree" timeout=10 {
          			_initializeEmptyTree();
          			var pathCache = variables.pageCache.getPathCache();
          			var q_source_files = variables.pageCache.getPageFileList(lastModified=true);
          			var added = 0;
          			var exists = {};
          			var deleted = [];
          
          
			loop query=q_source_files {
          				var addFile = false;
          				if (!structKeyExists(pathCache, q_source_files.path)){
          					addFile = true;
          				} else if (dateCompare(q_source_files.dateLastModified, pathCache[q_source_files.path]) eq 1) {
          					// do a timestamp comparison against the cache here
          					request.logger (text="source updated #q_source_files.path#");
          					addFile = true;
          				}
          
          
				if (addFile){
          					request.logger (text="Update pageCache, adding #q_source_files.path#");
          					variables.pageCache.addPage(

anyway I changed the request timeout from the default 50 up to 250 and I managed to get the
docs up and slowly limping on my core i7 -6700 machine

this is what I’m seeing, all the link titles are missing any ideas?

(all I wanted to do was to see why some categories didn’t work and why lists in the attributes descriptions aren’t working)

https://luceeserver.atlassian.net/browse/LD-108

bdw429s · March 23, 2018, 5:40pm

@Dominic_Watson

Dominic_Watson · March 24, 2018, 10:25am

I have fixed build failures on Linux. I think potentially some changes in Lucee have caused a behaviour change that broke the build.

Can you confirm fixed on Windows? (I have also commented on the JIRA tickets raised and merged the pull requests)

Zackster · March 24, 2018, 10:44am

build is still timing out, doesn’t crash any more, all the links are still showing up empty

Zackster · March 24, 2018, 4:12pm

I solved the speed issue, it now builds in 5s, which makes everything else much easier to debug

got rid of the windows specific regex for yaml extraction logic, i just force all line endings to unix style
the ancestor links were being duplicated for each child page of a parent, each page often had arrays with 4k items for ancestors and lineage, whether this was an existing problem or just a windows issue, i dunno
the forward and backward links were reprocessing all files multiple times, still broken but not slow
avoid empty see also links

a few things still aren’t working like the lists of functions and tags, I’m not sure if my changes break other things

Zackster · March 25, 2018, 8:26am

the docs will now be a bit more readable, lists weren’t being formatted all over the place
due to a missing new line in the markup before a lot of lists.

The local server is running nicely on windows now, it runs in 5s instead of 172s and uses less memory

recent improvements include

a markdown syntax checker which will catch these errors with lists in the future
a category checker which catches missing categories (currently disabled)
empty yaml variables are discarded as they cause empty list items
The Category page also now just lists all the categories, instead of hiding them behind the A-Z
removed the animation from the tags and function listing pages, it’s only fun the first time

The sub page elements (_attributes files mostly) were being added to the path map which wasn’t needed and was causing issues with the listing of tags and functions showing up empty. The problem was it was trying to render an _attributes page in the listing and they don’t have any metadata via yaml which the function or tag pages have.

This was partly due to the page type not being always extracted correctly
https://github.com/zspitzer/lucee-docs/blob/windows-debugging/api/data/PageReader.cfc#L7
https://github.com/zspitzer/lucee-docs/blob/windows-debugging/api/data/DocTree.cfc#L88

Could somebody try this out on linux/mac and see if everything is still working?
https://github.com/zspitzer/lucee-docs/tree/windows-debugging

Now that it’s working and running fast, next up I plan to look at:

in page editing of markdown elements, writing back to the file system directly
add a simple UI to assign categories and related (see also) items

Zackster · March 26, 2018, 2:03pm

i have a working prototype with inline editing!

the main tags and functions listings also now include a description

related links are also now working for tags and functions
pages inherit links from other pages which reference them

Dominic_Watson · March 28, 2018, 9:44am

Ha, this looks amazing. Docs building in 5s! Well done will check this out ASAP!

Zackster · March 28, 2018, 9:58am

Zackster · March 28, 2018, 10:02am

try clicking the edit icon in local server mode

Zackster · March 28, 2018, 10:35am

the new version is live

the new editing is only available when running locally, it saves your changes back to the local
file system, once you have finished you can commit to git and file a pull request

Dominic_Watson · April 3, 2018, 2:59pm

wowzers, awesome

Zackster · April 3, 2018, 3:16pm

it even auto fills the trycf markdown when there’s no examples

Zackster · April 3, 2018, 3:42pm

the local docs performance is sluggish because of cfdirectory scanning all the files
for each request for changes. Directory watcher can’t be used because you can’t use
mappings yet and it still does cfdirectory anyway…

once the mappings are supported for directory watcher and if directory
watcher gets updated to use a java watchService, the local edit mode will fly like the wind
https://luceeserver.atlassian.net/browse/LDEV-1776

modius · April 4, 2018, 12:28am

Would it be possible to implement a custom gateway for the Docs project rather than navigating/awaiting a change to Lucee server?

Zackster · April 4, 2018, 2:44am

it’s complicated, custom gateways are invisible in the admin which makes developing them more difficult

https://luceeserver.atlassian.net/browse/LDEV-1777

github.com

zspitzer/Lucee/blob/5.2/core/src/main/cfml/context/admin/services.gateway.list.cfm#L234


      
          						<th width="30%">#stText.Settings.gateway.state#</th>
          						<!---<th width="4%">#stText.Settings.DBCheck#</th>--->
          						<cfif type=='local'><th width="3%">&nbsp;</th></cfif>
          	
          
          
					</tr>
          				</thead>
          				<tbody>
          					<cfloop query="qry">						
          						<cfswitch expression="#qry.state#">
          							<cfcase value="running"><cfset css="Green"></cfcase>
          							<cfcase value="failed,stopped"><cfset css="Red"></cfcase>
          							<cfdefaultcase><cfset css="Yellow"></cfdefaultcase>
          						</cfswitch>
          						<tr>
          							<td>
          								<input type="checkbox" class="checkbox" name="row_#qry.currentrow#" value="#qry.currentrow#">
          							</td>
          							<td><input type="hidden" name="id_#qry.currentrow#" value="#qry.id#">#qry.id#</td>
          							<cfif IsSimpleValue(qry.driver)>
          								<td>#qry.driver#</td>

github.com

zspitzer/Lucee/blob/5.2/core/src/main/cfml/context/admin/services.gateway.cfm#L50


      
          	
          <!--- add driver to query --->
          <cfset QueryAddColumn(entries,"driver",array())>
          <cfloop query="entries">
              <cfloop collection="#drivers#" index="key" item="d">
              	<cfif 
          			(StructKeyExists(d,'getCFCPath')?d.getCFCPath() EQ entries.cfcPath:"" EQ entries.cfcPath)
          			and 
          			(StructKeyExists(d,'getClass')?d.getClass() EQ entries.class:"" EQ entries.class)
          			>
          			<cfset QuerySetCell(entries,"driver",d,entries.currentrow)>
                      
          		</cfif>
              </cfloop>
              
          </cfloop>
          
          

          
<cfswitch expression="#url.action2#">
          	<cfcase value="list"><cfinclude template="services.gateway.list.cfm"/></cfcase>
          	<cfcase value="create"><cfinclude template="services.gateway.create.cfm"/></cfcase>

i have an almost working directoryWatcher using a java watchService, but it would be great if somebody who is a better with java could jump in and figure out the last steps to get it working, i.e HELP WANTED

Zackster · April 9, 2018, 9:27am

I have changed the approach for this, I have added a parsed page cache and stopped scanning on every request. In local mode, there’s a new icon in the toolbar to trigger a rescan of the source files. Individual pages
are still always rendered from source file when requested in the browser.

the cog item opens a new interactive build menu, which lets you build the doco from a browser and shows real time updates as the various build processes progress.

I also did a lot of tuning on the performance of the rendering templates, it was a lot
easier to identify problems with the new javascript table sorting for tables in my performance.cfc debug template,
a documentation page now displays in around 30ms

Zackster · April 12, 2018, 4:26pm

I’ve got an experimental branch using flexmark instead of pegdown, it seems to be working.
https://github.com/zspitzer/lucee-docs/tree/LD-109-flexmark-parser

apart from being faster, the nice thing is flexmark is threadsafe, so we can build in parallel, using each

(minor aside, the argument validation for parallel and maxthreads could be stricter, you can mix them up and lucee doesn’t complain, ie parallel=8 and maxthreads=true doesn’t throw an error)

anyway, here are the numbers

/build_docs/all/ 93,561ms, 1 threads, 5.2.6.60 pegdown

/build_docs/all/ 70,040ms, 1 threads, 5.2.6.60 flexmark
/build_docs/all/ 46,532ms, 2 threads, 5.2.6.60 flexmark
/build_docs/all/ 32,314ms, 4 threads, 5.2.6.60 flexmark
/build_docs/all/ 29,809ms, 8 threads, 5.2.6.60 flexmark
/build_docs/all/ 34,434ms, 16 threads, 5.2.6.60 flexmark

it seems 4 threads is the sweet spot, my desktop doesn’t have a SSD, but it’s a grunty i7-6700,
there’s a lot of file i/o, so it might do better with a SSD

on my old laptop, a core i7-3632 with a ssd

/build_docs/all/ 199,992ms, 1 threads, 5.2.6.60 pegdown

/build_docs/all/ 183,835ms, 1 threads, 5.2.6.60 flexmark
/build_docs/all/ 116,618ms, 2 threads, 5.2.6.60 flexmark
/build_docs/all/ 81,473ms, 4 threads, 5.2.6.60 flexmark
/build_docs/all/ 61,755ms, 8 threads, 5.2.6.60 flexmark
/build_docs/all/ 59,213ms, 12 threads, 5.2.6.60 flexmark
/build_docs/all/ 59,972ms, 16 threads, 5.2.6.60 flexmark

last update: also doing the import in parallel on my desktop
/build_docs/all/ 22,296ms, 8 threads, 5.2.6.60

Zackster · April 15, 2018, 10:24am

I noticed there were quite a few typos in the #lucee documentation, so I added in a spell checker
https://luceeserver.atlassian.net/browse/LD-113

here are all the typos it detected which have been fixed