Live data from Hacker News

What can you do in 2k LOC of C?

h4ck3r.net

91–100 of 110 posts

Re: What can you do in 2k LOC of C?

#91

The core of my protobuf-decoding library upb ( https://github.com/haberman/upb/wiki ) is 3k SLOC of C. That includes * a hash table implementation * a reference-counted string type * a generic interface for doing tree traversals of protobuf data * the protobuf decoder itself (which implements the previous interface) * all the code to load proto descriptors (including bootstrapping the first one, which is necessary to…

It really says something about C that so many useful-but-small systems have a good chunk of code devoted to hash table implementations, atoms ("a reference-counted string type"), etc. When working with C, it sometimes makes sense to have bespoke data structures, but it always makes me think of Hanson's _C Interfaces and Implementations_, Greenspun's tenth rule, etc. My 1500loc C project also has a hash table implemen…

> It really says something about C that so many useful-but-small systems have a good chunk of code devoted to hash table implementations

For this speed-obsessed project, I wouldn't have it any other way:

  Benchmarking hash lookups in an integer-keyed hash table.

  Table size: 8, keys: 1-8 ====
  upb_inttable(seq): 410 M/s
  upb_inttable(rand): 334 M/s
  std::map(seq): 149 M/s
  std::map(rand): 170 M/s
  __gnu_cxx::hash_map(seq): 69.9 M/s
  __gnu_cxx::hash_map(rand): 68.0 M/s

  Table size: 64, keys: 1-64 ====
  upb_inttable(seq): 410 M/s
  upb_inttable(rand): 333 M/s
  std::map(seq): 32.4 M/s
  std::map(rand): 33.4 M/s
  __gnu_cxx::hash_map(seq): 67.9 M/s
  __gnu_cxx::hash_map(rand): 71.4 M/s

  Table size: 512, keys: 1-512 ====
  upb_inttable(seq): 407 M/s
  upb_inttable(rand): 330 M/s
  std::map(seq): 20.8 M/s
  std::map(rand): 17.2 M/s
  __gnu_cxx::hash_map(seq): 70.8 M/s
  __gnu_cxx::hash_map(rand): 64.6 M/s

  Table size: 64, keys: 1-32 and 10133-10164 ====
  upb_inttable(seq): 394 M/s
  upb_inttable(rand): 334 M/s
  std::map(seq): 32.0 M/s
  std::map(rand): 30.7 M/s
  __gnu_cxx::hash_map(seq): 71.3 M/s
  __gnu_cxx::hash_map(rand): 70.6 M/s

Re: What can you do in 2k LOC of C?

#92

Earlier quoted context omitted.

It really says something about C that so many useful-but-small systems have a good chunk of code devoted to hash table implementations, atoms ("a reference-counted string type"), etc. When working with C, it sometimes makes sense to have bespoke data structures, but it always makes me think of Hanson's _C Interfaces and Implementations_, Greenspun's tenth rule, etc. My 1500loc C project also has a hash table implemen…

> It really says something about C that so many useful-but-small systems have a good chunk of code devoted to hash table implementations For this speed-obsessed project, I wouldn't have it any other way: Benchmarking hash lookups in an integer-keyed hash table. Table size: 8, keys: 1-8 ==== upb_inttable(seq): 410 M/s upb_inttable(rand): 334 M/s std::map (seq): 149 M/s std::map (rand): 170 M/s __gnu_cxx::hash_map (seq…

Summarizing your results, your hash table implementation is between five and ten times faster than STL and G++ generic maps, because G++, like most C++ compilers, imposes a substantial abstraction penalty on the use of templates — even though in theory that's not necessary. Is that right?

Re: What can you do in 2k LOC of C?

#93
post #41

Earlier quoted context omitted.

Umm..13 people found it insightful/interesting? I don't know..

I can think of plenty of comments that were /actually/ insightful, though. This feels like, "oh, wow, you know how to use science! +1". sad.

I don't think the upvotes were intended as a positive judgment of you as a person, but rather as a positive judgment of the value of your comment. A sort of thank-you for doing the legwork so that the rest of us can simply glance at your comment instead of downloading nginx source and running sloccount ourselves.

Re: What can you do in 2k LOC of C?

#95
post #53

Earlier quoted context omitted.

Thanks! Partly Python's gotten more tolerable as a language, partly I'm doing more things needing libraries, partly I mostly code inside https://github.com/darius/halp these days and it doesn't have a Scheme mode so far. I do have a couple of recent Scheme projects up on github though -- optilamb and selfcentered. Very impressed with LuaJIT2, btw -- I'd like to do more with it.

Halp is fun. Thanks. Installed it already.

Glad you liked it!

Re: What can you do in 2k LOC of C?

#96
post #92

Earlier quoted context omitted.

> It really says something about C that so many useful-but-small systems have a good chunk of code devoted to hash table implementations For this speed-obsessed project, I wouldn't have it any other way: Benchmarking hash lookups in an integer-keyed hash table. Table size: 8, keys: 1-8 ==== upb_inttable(seq): 410 M/s upb_inttable(rand): 334 M/s std::map (seq): 149 M/s std::map (rand): 170 M/s __gnu_cxx::hash_map (seq…

Summarizing your results, your hash table implementation is between five and ten times faster than STL and G++ generic maps, because G++, like most C++ compilers, imposes a substantial abstraction penalty on the use of templates — even though in theory that's not necessary. Is that right?

In my last C++ program, 5 years ago, I did find it worthwhile to code a custom hashtable. Was a bit surprised. This was g++, probably an older version even then.

Re: What can you do in 2k LOC of C?

#97
post #92

Earlier quoted context omitted.

> It really says something about C that so many useful-but-small systems have a good chunk of code devoted to hash table implementations For this speed-obsessed project, I wouldn't have it any other way: Benchmarking hash lookups in an integer-keyed hash table. Table size: 8, keys: 1-8 ==== upb_inttable(seq): 410 M/s upb_inttable(rand): 334 M/s std::map (seq): 149 M/s std::map (rand): 170 M/s __gnu_cxx::hash_map (seq…

Summarizing your results, your hash table implementation is between five and ten times faster than STL and G++ generic maps, because G++, like most C++ compilers, imposes a substantial abstraction penalty on the use of templates — even though in theory that's not necessary. Is that right?

Yes, 5-10x faster. As to the reason, I can only speculate. I doubt it's template-imposed abstraction penalty per se, though it could be a result of having to write the algorithm in a generic way. In other words, if you did a sort of "manual instantiation" of the hash_map template, where you made the types specific but didn't change a thing besides that, I think you'd get the same performance as the templated algorithm.

I can't say for sure the reason for the difference. Previously I thought that hash_map might have been wasting time calculating a hash of the integer key (whereas I just use the key as the hash), but I just added another test that uses hash_map with an identity hash function and the performance was unchanged.

I bet that hash_map is using external chaining, whereas I'm using internal chaining which has better caching behavior. Besides that it's hard to say for sure why mine is so much faster.

Re: What can you do in 2k LOC of C?

#98
post #92

Earlier quoted context omitted.

Summarizing your results, your hash table implementation is between five and ten times faster than STL and G++ generic maps, because G++, like most C++ compilers, imposes a substantial abstraction penalty on the use of templates — even though in theory that's not necessary. Is that right?

Yes, 5-10x faster. As to the reason, I can only speculate. I doubt it's template-imposed abstraction penalty per se , though it could be a result of having to write the algorithm in a generic way. In other words, if you did a sort of "manual instantiation" of the hash_map template, where you made the types specific but didn't change a thing besides that, I think you'd get the same performance as the templated algorit…

Could be. I used internal chaining also in the program I mentioned in the other comment. And probably a power-of-2 size and another tweak or two -- I don't remember details.

Re: What can you do in 2k LOC of C?

#99
post #18

How much memory does the code the blog poster described leak? How does it respond to edge cases and invalid input? Finally, is it portable beyond one specific OS? Beyond POSIX or Windows-based systems? Those questions are especially pertinent in C.

Right! Coming up with standalone C means coming up with your own versions of hard-to-write simple "system calls" like memove. I debugged a memcopy (taken from Linux! 10 years ago) on a RISC processor - it had 12 bugs in a dozen lines of code. Not designed to run on RISC but shows how hard it can be to get this stuff right.

> simple "system calls" like memove

Minor, pet-peevy point: memmove(3) and memcpy(3) are not system calls. (http://en.wikipedia.org/wiki/System_call)

Re: What can you do in 2k LOC of C?

#100
post #99

Earlier quoted context omitted.

Right! Coming up with standalone C means coming up with your own versions of hard-to-write simple "system calls" like memove. I debugged a memcopy (taken from Linux! 10 years ago) on a RISC processor - it had 12 bugs in a dozen lines of code. Not designed to run on RISC but shows how hard it can be to get this stuff right.

> simple "system calls" like memove Minor, pet-peevy point: memmove(3) and memcpy(3) are not system calls. ( http://en.wikipedia.org/wiki/System_call )

Right; runtime library.

Often they are not even that; they appear to be but compilers can replace them bodily with super-optimized code so they get better benchmarks.

Post reply on HN