How big are PHP arrays (and values) really? • Hacker News Board

Re: How big are PHP arrays (and values) really?

#81

Post by cygx » Fri, Dec 16, 2011, 10:03 PM UTC

jeltz wrote: post #61

masklinn wrote: post #58

Earlier quoted context omitted.

OK (because that's not the case in CPython, I think)

Yeah, I too am pretty sure that CPython does not inline integers in the pointers. And from a quick glance at the source code I saw nothing such. Inlining integers in pointers by shifting up and adding 1 is a quite common trick though and I have seen it in more programming language implementations than MRI. I think at least some Prolog implementation and older versions of Spidermonkey (newer versions use a similar tri…

Inlining integers in pointers by shifting up and adding 1 is a quite common trick though and I have seen it in more programming language implementations than MRI

This trick was already used in Smalltalk-80, btw. A more recent variant of this is NaN tagging, made popular by LuaJIT.

Re: How big are PHP arrays (and values) really?

#82

Post by lysium » Fri, Dec 16, 2011, 10:23 PM UTC

obtu wrote: post #62

fhars wrote: post #7

Earlier quoted context omitted.

And once you relalize that they are really hashes you also realize that a fast and memory efficient representation will use at least nine machine words or 72 bytes per value: 1 + x for the pointer in the bucket array (you don't want 100% occupancy to avoid hash collisions), 2 for the cons cell of the list stored in the bucket (the second element of which will usually be NULL if you have a good hash function and low o…

That or specialize. Lua, Javascript, and PyPy all have more efficient storage of integer-keyed hashes. Lua's solution is particularly simple and could easily be transferred to PHP.

In PHP you can mix the types of the key, so that won't work.

Re: How big are PHP arrays (and values) really?

#83

Post by lysium » Fri, Dec 16, 2011, 10:31 PM UTC

lysium wrote: post #82

obtu wrote: post #62

Earlier quoted context omitted.

That or specialize. Lua, Javascript, and PyPy all have more efficient storage of integer-keyed hashes. Lua's solution is particularly simple and could easily be transferred to PHP.

In PHP you can mix the types of the key, so that won't work.

In another thread, dchest points out http://news.ycombinator.com/item?id=3360369 that Lua 5.0 allows mixed-typed keys, too, so that may work.

Re: How big are PHP arrays (and values) really?

#84

Post by [deleted] » Sat, Dec 17, 2011, 5:33 AM UTC

lysium wrote: post #82

obtu wrote: post #62

Earlier quoted context omitted.

That or specialize. Lua, Javascript, and PyPy all have more efficient storage of integer-keyed hashes. Lua's solution is particularly simple and could easily be transferred to PHP.

In PHP you can mix the types of the key, so that won't work.

[deleted]

Re: How big are PHP arrays (and values) really?

#85

Post by [deleted] » Sat, Dec 17, 2011, 5:34 AM UTC

lysium wrote: post #82

obtu wrote: post #62

Earlier quoted context omitted.

That or specialize. Lua, Javascript, and PyPy all have more efficient storage of integer-keyed hashes. Lua's solution is particularly simple and could easily be transferred to PHP.

In PHP you can mix the types of the key, so that won't work.

[deleted]

Re: How big are PHP arrays (and values) really?

#86

Post by wotnarg » Sat, Dec 17, 2011, 5:34 AM UTC

lysium wrote: post #82

obtu wrote: post #62

Earlier quoted context omitted.

That or specialize. Lua, Javascript, and PyPy all have more efficient storage of integer-keyed hashes. Lua's solution is particularly simple and could easily be transferred to PHP.

In PHP you can mix the types of the key, so that won't work.

You can mix types and specialize; HipHop does. The key is that the implementation needs to be able to switch array types on the fly and convert itself. In the common case, using a specialized array will be big wins in space and time, and in the uncommon case, it can revert to a generic version.

Re: How big are PHP arrays (and values) really?

#87

Post by josegonzalez » Sat, Dec 17, 2011, 7:33 AM UTC

rorrr wrote: post #74

wlievens wrote: post #26

Earlier quoted context omitted.

They are ordered by insertion. That's different from for instance objects in Javascript.

Not really $arr = array(); $arr[0] = 'cat'; $arr[2] = 'dog'; $arr[1] = 'fish'; krsort($arr); var_dump($arr); Outputs: array(3) { [2]=> string(3) "dog" [1]=> string(4) "fish" [0]=> string(3) "cat" }

Well you are explicitly sorting the array by keys, so its not like you are contradicting his sentence. Remove the krsort() call and your "not really" becomes "example".

Re: How big are PHP arrays (and values) really?

#88

Post by Gigablah » Sat, Dec 17, 2011, 9:39 AM UTC

maratd wrote: post #73

epochwolf wrote: post #70

Earlier quoted context omitted.

Isn't PHP-FPM a fastcgi process manager? I don't see any references to threading in it's documentation. I also recall the php documentation saying php is unsafe with multiple threads due to a large number of libraries that are not coded to be thread-safe.

> Isn't PHP-FPM a fastcgi process manager It is. > I don't see any references to threading Wow. I always thought it was threaded. My mistake. It seems it launches multiple child processes. There is some form of memory sharing going on, though. Looks like it's some sort of a hybrid. Now I'm just thoroughly confused ...

You can take a look at php-pcntl. http://www.php.net/manual/en/book.pcntl.php http://www.php.net/manual/en/function.pcntl-fork.php

Re: How big are PHP arrays (and values) really?

#89

Post by RobGR » Sun, Dec 18, 2011, 12:38 AM UTC

skeptical wrote: post #71

maratd wrote: post #66

Earlier quoted context omitted.

> Considering PHP's scope and its limitations (lacks things like threads for example) Please look up PHP-FPM. I use it in production, with great success. > If you're storing 100.000 integers on a PHP data structure you're most likely doing it wrong. What if I had a ton of price points and I needed to do statistical analysis? Well, those would be floats, but you get my point. PHP is capable of doing the work efficient…

How would you evoke such code? Squeezing it into an http reply? Calling it from PHP CLI? If doing it on your webapp, don't! You're doing it wrong. Otherwise why would you need php specifically? I mean I guess you could, but what advantage does it poses comparing to other languages available in pretty much every *nix system nowadays (perl, python)? For the record, I have a nice collection of downvotes for defending PH…

Firstly, I think the point of the demo of loading that many integers into the array, was to get a good measure of the memory usage per array item; by making the array big small one-time overhead memory usage is washed out in the average.

As to why you would want to do that in real life -- if you have written a large system with accompanying libraries, objects, or other infrastructure, then even if you pass of some sort of calculation to background or crontab tasks, you still might want to use your PHP code. I do this for some of the Drupal sites I work on - I setup drush commands that will do intensive calculations like most-related-article and etc to run behind the scenes have the results cached.

The best solution would be to improve PHP's memory usage so that the benefits may be had on page loads as well. I suspect (but have little hard evidence) that the size of PHP code and data structures impedes performance as threads and processes get pulled in and out of the CPU.

A secondary solution would be to take some of the libraries out there for doing specific operations, like scientific commputing and numerical libraries, and make them available to your code as a PHP extension. Obviously that doesn't solve as many problems as fixing PHP does.

Re: How big are PHP arrays (and values) really?

#90

Post by obtu » Mon, Dec 19, 2011, 9:53 AM UTC

lysium wrote: post #83

lysium wrote: post #82

Earlier quoted context omitted.

In PHP you can mix the types of the key, so that won't work.

In another thread, dchest points out http://news.ycombinator.com/item?id=3360369 that Lua 5.0 allows mixed-typed keys, too, so that may work.

Yes. You can specialise the implementation without breaking semantics. Lua, PyPy, and some Javascript VMs are examples of that.