Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How big are PHP arrays (and values) really?

nikic.github.com

Post reply on HN
Print view View original

61–70 of 90 posts

Re: How big are PHP arrays (and values) really?

#61

Post by jeltz »

masklinn wrote: post #58
jeltz wrote: post #57

Earlier quoted context omitted.

I assume so. If the integers fit in 31 or 63 bits, depending on your architecture, they are inlined in the pointer.

OK (because that's not the case in CPython, I think)

Yeah, I too am pretty sure that CPython does not inline integers in the pointers. And from a quick glance at the source code I saw nothing such.

Inlining integers in pointers by shifting up and adding 1 is a quite common trick though and I have seen it in more programming language implementations than MRI. I think at least some Prolog implementation and older versions of Spidermonkey (newer versions use a similar trick with doubles).

Re: How big are PHP arrays (and values) really?

#62

Post by obtu »

fhars wrote: post #7
rorrr wrote: post #3

PHP arrays are not really arrays, they are sort of hash-maps. You can do things like $arr = array(1 => 10, "1" => 11); Or even $arr = array('他妈的我的生活' => 5); But at the same time you can treat them as regular zero-based arrays. $arr = array(); $arr[] = 1; $arr[] = 2; $arr[] = 3; $arr[] = 'dog';

And once you relalize that they are really hashes you also realize that a fast and memory efficient representation will use at least nine machine words or 72 bytes per value: 1 + x for the pointer in the bucket array (you don't want 100% occupancy to avoid hash collisions), 2 for the cons cell of the list stored in the bucket (the second element of which will usually be NULL if you have a good hash function and low o…

That or specialize. Lua, Javascript, and PyPy all have more efficient storage of integer-keyed hashes. Lua's solution is particularly simple and could easily be transferred to PHP.

Re: How big are PHP arrays (and values) really?

#63

Post by masklinn »

jeltz wrote: post #61
masklinn wrote: post #58

Earlier quoted context omitted.

OK (because that's not the case in CPython, I think)

Yeah, I too am pretty sure that CPython does not inline integers in the pointers. And from a quick glance at the source code I saw nothing such. Inlining integers in pointers by shifting up and adding 1 is a quite common trick though and I have seen it in more programming language implementations than MRI. I think at least some Prolog implementation and older versions of Spidermonkey (newer versions use a similar tri…

> Inlining integers in pointers by shifting up and adding 1 is a quite common trick though and I have seen it in more programming language implementations than MRI.

Yeah, I know about it, I just did not think MRI had bothered with it anymore than CPython.

Re: How big are PHP arrays (and values) really?

#64

Post by masklinn »

scotty79 wrote: post #56
samdk wrote: post #23

Earlier quoted context omitted.

Python and Ruby both have separate arrays and hashes. They're completely different data structures in both cases. Python's dict type is a hash table like you'd expect. Python's list is a pointer-array-backed list. (it may inline ints/similar things--I don't remember if CPython does, and exact details are implementation-dependent), and raw arrays are in the standard library if you need them. From a very quick check, a…

Does python have standard implementation of odict already? It's nice that Python and Ruby have data structures that trade some features for some memory and/or performance gain.

> Does python have standard implementation of odict already?

Yeppers, since 2.7/3.1: http://docs.python.org/library/collections.html#collections.... http://docs.python.org/py3k/library/collections.html#collect...

Re: How big are PHP arrays (and values) really?

#65

Post by noselasd »

What does memory_get_usage() actually do ? Does it report the "heap" size assigned to the process, or does it use PHP internal counters for the allocated user data/variables ? A C malloc subsystem will assign a whole lot of virtual memory, in steps of pages, or more if it decides to attach a piece using mmap().

In order to make this test case relevant, I'd say one have to know what memory_get_usage() does - it's at least meaningless to determine the overhead of an array based on it, if for whatever reason creating the 1. PHP array in a program also initializes "big" memory pools that count towards the memory usage.

Re: How big are PHP arrays (and values) really?

#66

Post by maratd »

skeptical wrote: post #54
maratd wrote: post #2

As the author mentions, if you run into a use-case where you need to store 100000 integers in memory, then you should use one of the many alternative structures available. Some of them were explicitly designed to store integers in an efficient manner. Arrays weren't designed to store integers efficiently or anything else for that matter. They were designed to be fast and easy to use.

Considering PHP's scope and its limitations (lacks things like threads for example) I would say that is an extremely rare scenario. If you're storing 100.000 integers on a PHP data structure you're most likely doing it wrong.

> Considering PHP's scope and its limitations (lacks things like threads for example)

Please look up PHP-FPM. I use it in production, with great success.

> If you're storing 100.000 integers on a PHP data structure you're most likely doing it wrong.

What if I had a ton of price points and I needed to do statistical analysis? Well, those would be floats, but you get my point.

PHP is capable of doing the work efficiently, you just need to put a little thought in it first or you'll very quickly trash your box.

Re: How big are PHP arrays (and values) really?

#67

Post by maratd »

ktr wrote: post #59
dangrossman wrote: post #28

Earlier quoted context omitted.

That'd be why he said "arrays weren't designed to store integers efficiently".

From the comment: > then you should use one of the many alternative structures available. Some of them were explicitly designed to store integers in an efficient manner. From the article: > But if you do want to save memory you could consider using an SplFixedArray for large, static arrays. ... It basically does the same thing, but if you run it, you’ll notice that it uses “only” 5600640 bytes. That’s 56 bytes per el…

That is only one of the alternatives and in my opinion, not a very good one. I forget the exact details, but there is an extension by the guy who wrote igbinary that is specifically designed for this use-case.

Re: How big are PHP arrays (and values) really?

#68

Post by ezyang »

noselasd wrote: post #65

What does memory_get_usage() actually do ? Does it report the "heap" size assigned to the process, or does it use PHP internal counters for the allocated user data/variables ? A C malloc subsystem will assign a whole lot of virtual memory, in steps of pages, or more if it decides to attach a piece using mmap(). In order to make this test case relevant, I'd say one have to know what memory_get_usage() does - it's at l…

This is easy to check in the source. memory_get_usage() calls zend_memory_usage(), which accesses the size field on a global structure mm_heap, which is updated by PHP's memory allocation system (e.g. if you call *_zend_mm_alloc_int)

Re: How big are PHP arrays (and values) really?

#69

Post by vidarh »

masklinn wrote: post #63
jeltz wrote: post #61

Earlier quoted context omitted.

Yeah, I too am pretty sure that CPython does not inline integers in the pointers. And from a quick glance at the source code I saw nothing such. Inlining integers in pointers by shifting up and adding 1 is a quite common trick though and I have seen it in more programming language implementations than MRI. I think at least some Prolog implementation and older versions of Spidermonkey (newer versions use a similar tri…

> Inlining integers in pointers by shifting up and adding 1 is a quite common trick though and I have seen it in more programming language implementations than MRI. Yeah, I know about it, I just did not think MRI had bothered with it anymore than CPython.

false, true, nil, Symbol's and Fixnum's are all special cased with typetags in MRI.

Re: How big are PHP arrays (and values) really?

#70

Post by epochwolf »

maratd wrote: post #66
skeptical wrote: post #54

Earlier quoted context omitted.

Considering PHP's scope and its limitations (lacks things like threads for example) I would say that is an extremely rare scenario. If you're storing 100.000 integers on a PHP data structure you're most likely doing it wrong.

> Considering PHP's scope and its limitations (lacks things like threads for example) Please look up PHP-FPM. I use it in production, with great success. > If you're storing 100.000 integers on a PHP data structure you're most likely doing it wrong. What if I had a ton of price points and I needed to do statistical analysis? Well, those would be floats, but you get my point. PHP is capable of doing the work efficient…

Isn't PHP-FPM a fastcgi process manager? I don't see any references to threading in it's documentation. I also recall the php documentation saying php is unsafe with multiple threads due to a large number of libraries that are not coded to be thread-safe.
Post reply on HN