Building lucee documentation on windows and performance tuning


#1

I just finally got around to porting the docs build.sh and serve.sh over to batch files for windows

I hit the same error which the travis-ci is also hitting, it’s because the download folder doesn’t have a number for the sort order. doesn’t look like anything has changed recently, except maybe it’s using the 5.2.6.60 release?

https://travis-ci.org/lucee/lucee-docs/builds/357232550?utm_source=github_status&utm_medium=notification

[expression] error occurred while building the docs. Message: [invalid call of the function setSortOrder (/home/travis/build/lucee/lucee-docs/api/data/Page.cfc), first Argument (sortOrder) is of invalid type, can't cast String [] to a value of type [numeric]]. Detail: [].


#2

anyone familiar with the docs cfml code?

I think it’s doing way more recursive work than it needs to?

i don’t think page.getChildren() should be passed in to _calculateNextAndPreviousPageLinks again

anyway I changed the request timeout from the default 50 up to 250 and I managed to get the
docs up and slowly limping on my core i7 -6700 machine

this is what I’m seeing, all the link titles are missing any ideas?

(all I wanted to do was to see why some categories didn’t work and why lists in the attributes descriptions aren’t working)

https://luceeserver.atlassian.net/browse/LD-108


#3

@Dominic_Watson :point_up:


#4

I have fixed build failures on Linux. I think potentially some changes in Lucee have caused a behaviour change that broke the build.

Can you confirm fixed on Windows? (I have also commented on the JIRA tickets raised and merged the pull requests)


#5

build is still timing out, doesn’t crash any more, all the links are still showing up empty

image


#6

I solved the speed issue, it now builds in 5s, which makes everything else much easier to debug

  • got rid of the windows specific regex for yaml extraction logic, i just force all line endings to unix style
  • the ancestor links were being duplicated for each child page of a parent, each page often had arrays with 4k items for ancestors and lineage, whether this was an existing problem or just a windows issue, i dunno
  • the forward and backward links were reprocessing all files multiple times, still broken but not slow
  • avoid empty see also links

a few things still aren’t working like the lists of functions and tags, I’m not sure if my changes break other things


#7

the docs will now be a bit more readable, lists weren’t being formatted all over the place
due to a missing new line in the markup before a lot of lists.

The local server is running nicely on windows now, it runs in 5s instead of 172s and uses less memory

recent improvements include

  • a markdown syntax checker which will catch these errors with lists in the future
  • a category checker which catches missing categories (currently disabled)
  • empty yaml variables are discarded as they cause empty list items
  • The Category page also now just lists all the categories, instead of hiding them behind the A-Z
  • removed the animation from the tags and function listing pages, it’s only fun the first time

The sub page elements (_attributes files mostly) were being added to the path map which wasn’t needed and was causing issues with the listing of tags and functions showing up empty. The problem was it was trying to render an _attributes page in the listing and they don’t have any metadata via yaml which the function or tag pages have.

This was partly due to the page type not being always extracted correctly


Could somebody try this out on linux/mac and see if everything is still working?

Now that it’s working and running fast, next up I plan to look at:

  • in page editing of markdown elements, writing back to the file system directly
  • add a simple UI to assign categories and related (see also) items

#8

i have a working prototype with inline editing!

the main tags and functions listings also now include a description

  • related links are also now working for tags and functions
  • pages inherit links from other pages which reference them

#9

Ha, this looks amazing. Docs building in 5s! Well done :slight_smile: will check this out ASAP!


#10

#11

try clicking the edit icon in local server mode :slight_smile:


#12

the new version is live

the new editing is only available when running locally, it saves your changes back to the local
file system, once you have finished you can commit to git and file a pull request


#13

wowzers, awesome :slight_smile:


#14

it even auto fills the trycf markdown when there’s no examples :slight_smile:


#15

the local docs performance is sluggish because of cfdirectory scanning all the files
for each request for changes. Directory watcher can’t be used because you can’t use
mappings yet and it still does cfdirectory anyway…

once the mappings are supported for directory watcher and if directory
watcher gets updated to use a java watchService, the local edit mode will fly like the wind
https://luceeserver.atlassian.net/browse/LDEV-1776


#16

Would it be possible to implement a custom gateway for the Docs project rather than navigating/awaiting a change to Lucee server?


#17

it’s complicated, custom gateways are invisible in the admin which makes developing them more difficult

https://luceeserver.atlassian.net/browse/LDEV-1777


i have an almost working directoryWatcher using a java watchService, but it would be great if somebody who is a better with java could jump in and figure out the last steps to get it working, i.e HELP WANTED :slight_smile:


#20

I have changed the approach for this, I have added a parsed page cache and stopped scanning on every request. In local mode, there’s a new icon in the toolbar to trigger a rescan of the source files. Individual pages
are still always rendered from source file when requested in the browser.

image

the cog item opens a new interactive build menu, which lets you build the doco from a browser and shows real time updates as the various build processes progress.

image

I also did a lot of tuning on the performance of the rendering templates, it was a lot
easier to identify problems with the new javascript table sorting for tables in my performance.cfc debug template,
a documentation page now displays in around 30ms


#21

I’ve got an experimental branch using flexmark instead of pegdown, it seems to be working.

apart from being faster, the nice thing is flexmark is threadsafe, so we can build in parallel, using each

(minor aside, the argument validation for parallel and maxthreads could be stricter, you can mix them up and lucee doesn’t complain, ie parallel=8 and maxthreads=true doesn’t throw an error)

anyway, here are the numbers

/build_docs/all/ 93,561ms, 1 threads, 5.2.6.60 pegdown

/build_docs/all/ 70,040ms, 1 threads, 5.2.6.60 flexmark
/build_docs/all/ 46,532ms, 2 threads, 5.2.6.60 flexmark
/build_docs/all/ 32,314ms, 4 threads, 5.2.6.60 flexmark
/build_docs/all/ 29,809ms, 8 threads, 5.2.6.60 flexmark
/build_docs/all/ 34,434ms, 16 threads, 5.2.6.60 flexmark

it seems 4 threads is the sweet spot, my desktop doesn’t have a SSD, but it’s a grunty i7-6700,
there’s a lot of file i/o, so it might do better with a SSD

on my old laptop, a core i7-3632 with a ssd

/build_docs/all/ 199,992ms, 1 threads, 5.2.6.60 pegdown

/build_docs/all/ 183,835ms, 1 threads, 5.2.6.60 flexmark
/build_docs/all/ 116,618ms, 2 threads, 5.2.6.60 flexmark
/build_docs/all/ 81,473ms, 4 threads, 5.2.6.60 flexmark
/build_docs/all/ 61,755ms, 8 threads, 5.2.6.60 flexmark
/build_docs/all/ 59,213ms, 12 threads, 5.2.6.60 flexmark
/build_docs/all/ 59,972ms, 16 threads, 5.2.6.60 flexmark

last update: also doing the import in parallel on my desktop
/build_docs/all/ 22,296ms, 8 threads, 5.2.6.60


#22

I noticed there were quite a few typos in the #lucee documentation, so I added in a spell checker
https://luceeserver.atlassian.net/browse/LD-113

here are all the typos it detected which have been fixed