This is great! Inspired me to make a little script in spark-shell: > val words = oxforddict. filter{case s => s.exists(_.isLetter) && s.forall(Character.isUpperCase(_))}. distinct. map((_.toLowerCase,1)) > def lcp(a:String,b:String) = { a.zip(b).takeWhile(Function.tupled(_ == _)).map(_._1).mkString } > implicit val wordOrdering = new Ordering[String] { > override def compare(a:String, b:String) = { val lcpIdx = lcp(a…
The first thing that comes to mind is skipping past letters that don't appear very often at that point in a word with the current prefix--or to be more granular, you could change your counter (here g) to a float and somehow weight each letter by how rarely it occurs after the letters you've established so far. So if you've currently established that the first four letters are "pro," and "z" almost never occurs after those letters, "z" might be given a weight above 1 so the counter skips right past it.