How big are PHP arrays (and values) really? • Hacker News Board

Re: How big are PHP arrays (and values) really?

#51

Post by masklinn » Fri, Dec 16, 2011, 11:25 AM UTC

lvh wrote: post #43

masklinn wrote: post #35

Earlier quoted context omitted.

> AFAIC Python and Ruby are also memory hogs. Even more so in some areas, for instance a Python `int` is not a machine integer but a full-blown object.

It's only always a full-blown object in CPython. Smarter implementations, notably PyPy, will do escape analysis, and never actually end up allocating those objects. (My point is that not having unboxed types does not imply being a memory hog. You just need a smarter implementation.)

> Smarter implementations, notably PyPy, will do escape analysis, and never actually end up allocating those objects.

Objects in a collection (which is what we're talking about here) escape kind-of by default.

Until type-specialized collections are merged in PyPy (if they are not yet), it'll have the same issue as CPython.

Re: How big are PHP arrays (and values) really?

#52

Post by jeltz » Fri, Dec 16, 2011, 11:26 AM UTC

samdk wrote: post #23

fhars wrote: post #11

Earlier quoted context omitted.

Since PHP arrays are within a factor of two for the theoretical optimum for a dynamically typed language that unifies arrays and hashes, all these languages (add javascript to the list) should have roughly comparable behaviour unless they do some ugly special casing for hashes where all keys are integers. [edit: see the comment by InfernalH for a measurement that indicates that ruby seems to do this.] You have to loo…

Python and Ruby both have separate arrays and hashes. They're completely different data structures in both cases. Python's dict type is a hash table like you'd expect. Python's list is a pointer-array-backed list. (it may inline ints/similar things--I don't remember if CPython does, and exact details are implementation-dependent), and raw arrays are in the standard library if you need them. From a very quick check, a…

Good info about Python, but you are incorrect about Ruby. Hashes and arrays are totally different in the Ruby implementations I know of.

EDIT: Sorry, I misread you. Ruby MRI and CPython are indeed similar.

At least in Ruby MRI (the mainline) arrays are implemented as a struct with a size and a pointer to a normal C array which contains the object references (references in MRI are pointers to object structs which use the lower bits to inline integers of Hashes in MRI I have not looked that much into but I believe they are a hash tables which in ruby 1.9 retain insertion order using pointers like a singly linked list.

Re: How big are PHP arrays (and values) really?

#53

Post by cygx » Fri, Dec 16, 2011, 11:29 AM UTC

legooolas wrote: post #49

cygx wrote: post #47

Earlier quoted context omitted.

Type-punning through unions is already legal in C99, but there's a known error in Annex J, listing it incorrectly as unspecified behaviour. See http://stackoverflow.com/a/8513748/48015

I stand corrected :/

Don't feel bad about it - the C standard can be quite subtle, and I've been known to spread lies about it as well.

Things about which I have stumbled somewhat recently:

* restrict-qualified pointer-to-const parameters do not guarantee that the pointed-to object won't be modified as restrict only applies if the pointer is actually used to access the object, which calling code can't know (ie restrict only enables optimizations in the called code and not reordering in calling code)

* functions with differently qualified, but otherwise compatible parameter types have compatible type (which is only mentioned in the last, parenthesized sentence of section 6.7.5.3)

Re: How big are PHP arrays (and values) really?

#54

Post by skeptical » Fri, Dec 16, 2011, 11:51 AM UTC

maratd wrote: post #2

As the author mentions, if you run into a use-case where you need to store 100000 integers in memory, then you should use one of the many alternative structures available. Some of them were explicitly designed to store integers in an efficient manner. Arrays weren't designed to store integers efficiently or anything else for that matter. They were designed to be fast and easy to use.

Considering PHP's scope and its limitations (lacks things like threads for example) I would say that is an extremely rare scenario. If you're storing 100.000 integers on a PHP data structure you're most likely doing it wrong.

Re: How big are PHP arrays (and values) really?

#55

Post by masklinn » Fri, Dec 16, 2011, 11:52 AM UTC

jeltz wrote: post #52

samdk wrote: post #23

Earlier quoted context omitted.

Python and Ruby both have separate arrays and hashes. They're completely different data structures in both cases. Python's dict type is a hash table like you'd expect. Python's list is a pointer-array-backed list. (it may inline ints/similar things--I don't remember if CPython does, and exact details are implementation-dependent), and raw arrays are in the standard library if you need them. From a very quick check, a…

Good info about Python, but you are incorrect about Ruby. Hashes and arrays are totally different in the Ruby implementations I know of. EDIT: Sorry, I misread you. Ruby MRI and CPython are indeed similar. At least in Ruby MRI (the mainline) arrays are implemented as a struct with a size and a pointer to a normal C array which contains the object references (references in MRI are pointers to object structs which use…

> Good info about Python, but you are incorrect about Ruby. Hashes and arrays are totally different in the Ruby implementations I know of.

From its context, I'm guessing samdk is saying:

> Ruby's hash and array implementations are similar [to Python's dict and list]

not that they're similar to one another, which would make absolutely no sense considering his comment starts with:

> Python and Ruby both have separate arrays and hashes.

so I'd say you agree with him and misread his comment.

Re: How big are PHP arrays (and values) really?

#56

Post by scotty79 » Fri, Dec 16, 2011, 12:08 PM UTC

samdk wrote: post #23

fhars wrote: post #11

Earlier quoted context omitted.

Since PHP arrays are within a factor of two for the theoretical optimum for a dynamically typed language that unifies arrays and hashes, all these languages (add javascript to the list) should have roughly comparable behaviour unless they do some ugly special casing for hashes where all keys are integers. [edit: see the comment by InfernalH for a measurement that indicates that ruby seems to do this.] You have to loo…

Python and Ruby both have separate arrays and hashes. They're completely different data structures in both cases. Python's dict type is a hash table like you'd expect. Python's list is a pointer-array-backed list. (it may inline ints/similar things--I don't remember if CPython does, and exact details are implementation-dependent), and raw arrays are in the standard library if you need them. From a very quick check, a…

Does python have standard implementation of odict already?

It's nice that Python and Ruby have data structures that trade some features for some memory and/or performance gain.

Re: How big are PHP arrays (and values) really?

#57

Post by jeltz » Fri, Dec 16, 2011, 12:25 PM UTC

masklinn wrote: post #37

InfernalH wrote: post #10

Earlier quoted context omitted.

I tried with a "pure" Array - for Ruby it's 1Mb for 100 000 elements EDIT: and for Hash with h[i] = i, it's ~6Mb

Does that account for the size taken by the integers themselves?

I assume so. If the integers fit in 31 or 63 bits, depending on your architecture, they are inlined in the pointer.

Re: How big are PHP arrays (and values) really?

#58

Post by masklinn » Fri, Dec 16, 2011, 1:06 PM UTC

jeltz wrote: post #57

masklinn wrote: post #37

Earlier quoted context omitted.

Does that account for the size taken by the integers themselves?

I assume so. If the integers fit in 31 or 63 bits, depending on your architecture, they are inlined in the pointer.

OK (because that's not the case in CPython, I think)

Re: How big are PHP arrays (and values) really?

#59

Post by ktr » Fri, Dec 16, 2011, 1:38 PM UTC

dangrossman wrote: post #28

mbell wrote: post #4

Earlier quoted context omitted.

I wouldn't call using 56 bytes to store 8 bytes of actual data efficient.

That'd be why he said "arrays weren't designed to store integers efficiently".

From the comment:

> then you should use one of the many alternative structures available. Some of them were explicitly designed to store integers in an efficient manner.

From the article:

> But if you do want to save memory you could consider using an SplFixedArray for large, static arrays. ... It basically does the same thing, but if you run it, you’ll notice that it uses “only” 5600640 bytes. That’s 56 bytes per element ...

EDIT: formatting.

Re: How big are PHP arrays (and values) really?

#60

Post by obtu » Fri, Dec 16, 2011, 1:54 PM UTC

nestlequ1k wrote: post #5

Interesting. Wonder how this compares to python and ruby memory handling.

Python has lists and dicts, and lists don't get the per-item overhead of GC-ed keys. PyPy does better than CPython and implements specialised, low-overhead storage for some dicts, lists and sets of uniform type (in 1.6, nightly builds, and a branch respectively).

http://morepypy.blogspot.com/2011/10/more-compact-lists-with...