Parallel Stage 2

[dmitry-shechtman]

Hey,

I added a branch that contains a tentative parallelization of Stage 2. I didn't see a significant improvement in CommonBenchMark, so maybe this isn't worth pursuing.

Please let me know what you think.

Thanks

[Knagis]

Did you actually get any significant performance improvement after the latest changes? The reason why I did not pursue the parallel approach is that for all realistic inputs (that are not very rare 10MB books) it actually slowed things down.

Why is Partition creating clones of the settings (since it is not done in non-parallel approach) and reference map?

p.s. features like this should be kept in a separate branch before completion, to leave master clean and ready for quick releases, if such need arises.

[dmitry-shechtman]

I got some small improvements, but nothing out of the ordinary so far. emphasis.txt is down to 15%, but that's hardly a realistic input. I'd love to put my hands on some of those 10MB books you mentioned.

Cloning the settings was my attempt at creating memory locality. I'll try putting parser cache in Subject instead. As for ReferenceMap, cloning actually seems to be slowing things down.

Another approach I want to try is a deeper dive in the syntax tree.

Of course I'm not going to merge to master before I'm sure I got it 100% right. I am quite a perfectionist myself, you know ;)

[Knagis]

Of course I'm not going to merge to master before I'm sure I got it 100% right. I am quite a perfectionist myself, you know ;)

But most of the commits are made against the master branch and not the parallel branch...

Some more concerns about performance - probably the most common size for markdown text is under few kilobytes - does the paralellization help with the small documents?

Also on a webserver parallel execution might even be frowned upon because the webserver is using all the cores to process requests and thus the overhead from the parallelization in markdown parser is unwanted.

Overall if the addition of one extra core to the parser does not result in at least average 30% increase in performance it might not be worth it - let the application parallelize parsing multiple documents instead.

[dmitry-shechtman]

I am so sorry. I was working in a local parallel branch, so I presumed I was pushing to remote parallel.

Let me know if you want me to revert the parallelization-related commits.

[Knagis]

I did a force-push 😄 of the master to restore it to before those commits. You might need to do something manual to get your local master branch back in sync and so that you do not push those commits again into master.
I'm no expert but it could be that the easiest way is to delete the local master branch and then checkout the origin/master to recreate it.

[dmitry-shechtman]

I don't actually have a local master, so no damage here.

I did add a deeper child count, but am still seeing little to no improvement (even not in emphasis.txt, which was measurement impurity).

I suspect this might have to do with my laptop overheating. May I ask you to please try it on your machine? If there are no real gains, I guess we should call it quits.

The upside is that I modified MarkdownCompare to output nice pipe tables. Would you like me to do a PR on that?

P.S. I didn't manage to find any sufficiently large inputs, and I believe the smallish ones aren't worth the overhead.