How to Split Sentences (2014)
tech.grammarly.com
How to Split Sentences (2014)
1–10 of 13 posts
Re: How to Split Sentences (2014)
#2I did an analysis of different sentence segmentation tools when I was working on my own rule-based segmenter. The results can be found in this README (https://github.com/diasks2/pragmatic_segmenter).
I think this blog post almost hits on the key in the middle - in my opinion it is important to test (all of) the edge cases. The problem with most corpora typically used to test segmenters is that 80-90% of the sentences are the same (i.e. a regular sentence ending in a period). Thus if a segmenter just simply split the sentence at every period it would still show a 80-90% accuracy rate. This is why I am trying to develop a standardized set of edge cases: https://github.com/diasks2/pragmatic_segmenter#the-golden-ru...
Re: How to Split Sentences (2014)
#3Self-promotion: I wrote an open-source sentence splitter tool that outperforms the state of the art on the "standard split". It is also very fast.
http://sonny.cslu.ohsu.edu/~gormanky/blog/simpler-sentence-b... (link to GitHub repo in post)
Re: How to Split Sentences (2014)
#4I've just added this kind of support to node-summary (https://github.com/jbrooksuk/node-summary) which seems to make a bit of a positive difference under the tests.
Re: How to Split Sentences (2014)
#5I did an analysis of different sentence segmentation tools when I was working on my own rule-based segmenter. The results can be found in this README ( https://github.com/diasks2/pragmatic_segmenter ). I think this blog post almost hits on the key in the middle - in my opinion it is important to test (all of) the edge cases. The problem with most corpora typically used to test segmenters is that 80-90% of the sentenc…
Great work. I love the "Golden Rules" list you compiled. It seems like teams develop their NLP systems without sharing a common training set which leaves some teams without testing things like the "a.m. / p.m." thing.
Re: How to Split Sentences (2014)
#6I did an analysis of different sentence segmentation tools when I was working on my own rule-based segmenter. The results can be found in this README ( https://github.com/diasks2/pragmatic_segmenter ). I think this blog post almost hits on the key in the middle - in my opinion it is important to test (all of) the edge cases. The problem with most corpora typically used to test segmenters is that 80-90% of the sentenc…
Worth taking a look at the unicode sentence segmentation algorithm rules: http://unicode.org/reports/tr29/#Sentence_Boundaries
Also at the CLDR sentence break supressions: http://unicode.org/cldr/trac/browser/tags/release-27-0-1/com...
If your rules treat an edge case that the above don't it'd probably be worth trying to suggest improvements to the unicode rules or the locale-specific ones.
Re: How to Split Sentences (2014)
#7is anyone aware of a sentence segmenter for poorly written english that is missing some punctation? like from chat sessions? it could be useful for normal sentence segmentation. i.e. if you forget about the punctuation, can you detect the boundaries of the sentence anyway.
Re: How to Split Sentences (2014)
#8I did an analysis of different sentence segmentation tools when I was working on my own rule-based segmenter. The results can be found in this README ( https://github.com/diasks2/pragmatic_segmenter ). I think this blog post almost hits on the key in the middle - in my opinion it is important to test (all of) the edge cases. The problem with most corpora typically used to test segmenters is that 80-90% of the sentenc…
Looks good!
Have you tried to evaluate your splitter on some other data, on this "typically used corpora"? The evaluation quality looks too optimistic - 98% / 100% quality means you made your code to work on your examples, but by using only a set of standartized tests you can't check:
* how broad is the coverage - there are other edge cases in real world, it may be impossible to cover them all;
* that the splitter doesn't make mistakes for real-world "regular" sentences (80-90% of sentences which are "the same").
The example set looks very good, and it looks like a good way to compare other sentence splitters. But it is not fair to provide evaluation metrics on the examples you used to develop your sentence splitter.
Re: How to Split Sentences (2014)
#9I did an analysis of different sentence segmentation tools when I was working on my own rule-based segmenter. The results can be found in this README ( https://github.com/diasks2/pragmatic_segmenter ). I think this blog post almost hits on the key in the middle - in my opinion it is important to test (all of) the edge cases. The problem with most corpora typically used to test segmenters is that 80-90% of the sentenc…
Great work. I love the "Golden Rules" list you compiled. It seems like teams develop their NLP systems without sharing a common training set which leaves some teams without testing things like the "a.m. / p.m." thing.
Not only there is little sharing, it is very focused on English.
Re: How to Split Sentences (2014)
#10I did an analysis of different sentence segmentation tools when I was working on my own rule-based segmenter. The results can be found in this README ( https://github.com/diasks2/pragmatic_segmenter ). I think this blog post almost hits on the key in the middle - in my opinion it is important to test (all of) the edge cases. The problem with most corpora typically used to test segmenters is that 80-90% of the sentenc…
Great work. I love the "Golden Rules" list you compiled. It seems like teams develop their NLP systems without sharing a common training set which leaves some teams without testing things like the "a.m. / p.m." thing.
For comparability, most people use the Penn Treebank-III WSJ data. Sections 03-06 are test, the remaining sections are train/dev.
Most methods are based on some sort of simple feature templates and machine learning, so they should generalize relatively well to a wide variety of languages, IMO.