The key innovation of Scryer Prolog in this respect is that the system
internally, i.e., within Rust, represents each string as a sequence of raw bytes (UTF-8 encoded, 0-terminated), and
to Prolog programs, the string appears as if it were a
list of characters (i.e., atoms of length 1). So, no conversion is necessary at all: The compact internal representation is used whenever possible, and Prolog programs only see lists of characters all of the time, making common predicates such as append/3 and length/2, and most importantly DCGs, automatically usable for reasoning about strings.
For example, the string "abc" takes only 4 bytes in this representation, and the unification "abc" = [a,b,c] succeeds. Indeed, we have:
?- write_canonical("abc").
'.'(a,'.'(b,'.'(c,[]))) true.
On 64-bit systems, a conventional internal representation of "abc" as the compound term .(a, .(b, .(c, []))) takes 8 bytes per functor (each '.'/2 takes 8 bytes, i.e., 1 cell in the WAM), 8 bytes per character (as a pointer to the atom table, again 1 cell in the WAM), and 8 bytes for each tail. All Prolog systems before Scryer Prolog use 16 to 24
times as much memory to represent a list of characters. As of recent, Trealla Prolog also uses the efficient representation, at least for
fully instantiated strings. Scryer Prolog can also represent
partial strings such as [a,b,c|Ls] with the efficient encoding.