Tell HN: Python's print function is thread-unsafe

#1

Post by kerneloops » Mon, Jan 01, 2024, 8:31 AM UTC

Some days ago I found that Python's print function is by default thread-unsafe, as the underlying sys.stdout is by default a TextIOWrapper, which is not thread safe. That means, for the code below

  from threading import Thread
  def target(x):
      for _ in range(100000): print(x)
  Thread(target=target, args=['a' * 10]).start()
  Thread(target=target, args=['b' * 10]).start()

Python could print out not only interleaved bytes (aaabbbabab), but also null bytes and uninitialized memory, thanks to non-synchronized buffering.

Re: Tell HN: Python's print function is thread-unsafe

#2

Post by kerneloops » Mon, Jan 01, 2024, 8:35 AM UTC

A survey of some other languages:

C printf: MT-Safe locale.

C++ std::cout: safe, unless you call sync_with_stdio(false).

JVM System.out.println: safe in common JVMs.

C# Console.WriteLine: safe.

Go fmt.Printf: safe.

Rust println!: safe.

Ruby puts: safe.

So it seems that Python is the outlier here.

Re: Tell HN: Python's print function is thread-unsafe

#3

Post by cratermoon » Mon, Jan 01, 2024, 9:11 AM UTC

Why Threads Are a Bad Idea: https://news.ycombinator.com/item?id=17297325

The problem with threads: https://ieeexplore.ieee.org/document/1631937

Re: Tell HN: Python's print function is thread-unsafe

#4

Post by antman » Mon, Jan 01, 2024, 9:52 AM UTC

See this example:

https://superfastpython.com/thread-safe-print-in-python/

Python logging is thread safe if you want to use it for output

Re: Tell HN: Python's print function is thread-unsafe

#5

Post by eesmith » Mon, Jan 01, 2024, 10:02 AM UTC

I was able to reproduce your failure (using Python 3.10) after a couple of runs:

  % grep -v '^aaaaaaaaaa$' kerneloops.out | grep -v '^bbbbbbbbbb$' | od -x
  0000000      8130    0db3    0001    0000    bd30    0dad    0001    0000
  *
  0006420      000a
  0006421

I was surprised as I expected possible interleaved bytes but not output other than "a"s, "b"s and new-lines, as I thought the GIL would come to the rescue.

Re: Tell HN: Python's print function is thread-unsafe

#6

Post by ansgri » Mon, Jan 01, 2024, 1:57 PM UTC

cratermoon wrote: post #3

Why Threads Are a Bad Idea: https://news.ycombinator.com/item?id=17297325 The problem with threads: https://ieeexplore.ieee.org/document/1631937

Given threads are here to stay, thread-unsafe print is a really bad idea. Couldn’t even imagine it can crash instead of interleave, in Python of all things.

Re: Tell HN: Python's print function is thread-unsafe

#7

Post by vitaut » Mon, Jan 01, 2024, 4:49 PM UTC

kerneloops wrote: post #2

A survey of some other languages: C printf: MT-Safe locale. C++ std::cout: safe, unless you call sync_with_stdio(false). JVM System.out.println: safe in common JVMs. C# Console.WriteLine: safe. Go fmt.Printf: safe. Rust println!: safe. Ruby puts: safe. So it seems that Python is the outlier here.

std::cout is thread-safe but because of the concatenation-based API parts of the message may interleave. C++23 std::print is safe (and the output doesn't interleave).

See https://vitaut.net/posts/2023/print-in-cpp23/.