Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redis on the Raspberry Pi: Adventures in unaligned lands

antirez.com

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Redis on the Raspberry Pi: Adventures in unaligned lands

#41

Post by throwaway000002 »

pm215 wrote: post #40
throwaway000002 wrote: post #32

I'm probably the only weirdo that thinks this, but if you support byte-addressing you'd better as well be happy with byte-alignment. Atomics being the only place where it's reasonable to be different. Which brings me to padding. I wonder what percentage of memory of the average 64-bit user's system is padding? I'm afraid of the answer. The heroes of yesteryear could've coded miracles in the ignored spaces in our data…

Alignment requirements are and have historically been very common -- you can see them on the PDP-11, the 680x0, and so on. It's only because a few very popular architectures like x86 have had very loose or no alignment requirements that we've ended up with a lot of code that assumes there is no alignment requirement, and this has dragged other architectures down the "we need to support this" path. If your architectur…

I can understand the historical requirements for alignment, the necessary transistors, what not. But, much like branch-delay slots, there is no modern reason to expose this to the programmer. Of course, I gave an exception to atomics, but if you will, they're like memory-mapped communication, and now that all I/O is memory-mapped, with no concept of ports, the (ordering) semantics of memory access becomes really important.

I'm also the weirdo that feels process isolation, memory management, and I/O mechanisms need a rethink. But that's something that would take me forever to get into.

One thing I will say, though, is alignment issues "infect" everything. Assume your architecture doesn't allow misaligned access. Now, all your data has to be naturally aligned. Your structs now have to be aligned to the alignment of the largest sub-structure within them. This is all because code is alignment sensitive. Given a pointer to a struct, generic code is unnecessarily larger. Any why would we care? Communication, of course. If we're exchanging data between systems then idiosyncrasies such as this suddenly become globally visible.

Endian-ness must be little. Byte-aligment a non-issue, and network-bit order should be from bit zero up, with any upper layer need, say for cut-through forwarding, expressed as a data ordering requirement, so for example an IP4 address is not a blind 32-bit word, but specifies the structure of those 32-bits.

Re: Redis on the Raspberry Pi: Adventures in unaligned lands

#42

Post by jandrese »

IgorPartola wrote: post #11
drewg123 wrote: post #10

I fondly remember unaligned access faults "back in the day" with FreeBSD/alpha. We implemented a fixup for applications, but not for the kernel. I seem to recall that even though x86 could deal with unaligned accesses, it caused minor performance problems, so fixing alignment issues on alpha would benefit x86 as well. Most (definitely not all) of the mis-alignment problems were in the network stack, and were centered…

Why in god's name did they make it 14?!

It's all they needed. 6 bytes per address, and 2 more bytes to mark the protocol. Back in the 70s and 80s memory was very expensive and developers bent over backwards to save bytes everywhere. This is also why IP addresses are only 32 bits long, even though they knew that it wouldn't be enough if the protocol went global.

Hindsight is 20/20, and a lot of times people don't appreciate the constraints these old systems had. This was being developed decade before the Commodore 64 came out with its luxurious 64 kilobytes of memory (39k usable).

Re: Redis on the Raspberry Pi: Adventures in unaligned lands

#43

Post by pm215 »

throwaway000002 wrote: post #41
pm215 wrote: post #40

Earlier quoted context omitted.

Alignment requirements are and have historically been very common -- you can see them on the PDP-11, the 680x0, and so on. It's only because a few very popular architectures like x86 have had very loose or no alignment requirements that we've ended up with a lot of code that assumes there is no alignment requirement, and this has dragged other architectures down the "we need to support this" path. If your architectur…

I can understand the historical requirements for alignment, the necessary transistors, what not. But, much like branch-delay slots, there is no modern reason to expose this to the programmer. Of course, I gave an exception to atomics, but if you will, they're like memory-mapped communication, and now that all I/O is memory-mapped, with no concept of ports, the (ordering) semantics of memory access becomes really impo…

Even today, allowing unaligned accesses is still not free -- there is an implementation cost in transistors and in design complexity. There's a tradeoff here, as usual. There are a lot of places with a CPU architecture where there's a choice of "do we handle this in hardware, at the cost of having to have more hardware, or do we say it's software's job to deal with this, and hardware provides either nothing or just some helpful tools". You can see this for instance in whether software has to perform icache/dcache maintenance vs the CPU doing a lot of snooping to present the illusion of a completely coherent system; in whether hypervisor virtual machine switching is done with a single "switch all my state" operation on by letting hypervisor software switch register state itself; and in many other places. x86 has in my view generally ended up on the "handle things in hardware and make software's life easier", which it's been able to do because its natural territory is desktop/server where extra transistors don't hurt much. Other architectures tend towards different points on this spectrum because their constraints differ -- in embedded systems the extra power and are cost of more transistors can really matter. "Tend to prefer that software do something" is also a strand of the original RISC philosophies.

Practically speaking, the world is not going to converge on a single endianness or on a no-alignment-restrictions setup any time soon, so we have to deal with the world as it is. If you're programming in a sensible high-level language, it will deal with this kind of low-level nit for you. If you're programming in a low-level language (like C), well, I think you wouldn't be doing that if you didn't have fun at some level in feeling like you had a mastery of the low-level nits :-)

Re: Redis on the Raspberry Pi: Adventures in unaligned lands

#44

Post by parthdesai »

striking wrote: post #25
parthdesai wrote: post #21

Earlier quoted context omitted.

your username will appear green till you get a certain amount of karma/upvotes.

I thought it was based on time.

hmm not sure. My username appeared green till i got 50/100 upvotes. I think that makes sense since the number upvotes on HN usually indicates that the person makes valid comments and is not trolling.

Re: Redis on the Raspberry Pi: Adventures in unaligned lands

#45

Post by bbatha »

amelius wrote: post #12

Could Rust's typesystem catch unaligned pointer dereferences?

Sort of, Rust is supposed to make references to packed structure members unsafe, but currently doesn't. An RFC was accepted to change the behavior but it has not been fully implemented. Here's the tracking issue: https://github.com/rust-lang/rust/issues/27060

Re: Redis on the Raspberry Pi: Adventures in unaligned lands

#46

Post by msarnoff »

Recently I've been doing a lot of low-level work with ARMv7-M microcontrollers (specifically, NXP's Kinetis Cortex-M4 chips) and was quite pleased to find out that they are pretty lenient about unaligned accesses. To quote from the ARM Cortex-M4 Processor Technical Reference Manual:

"Unaligned word or halfword loads or stores add penalty cycles. A byte aligned halfword load or store adds one extra cycle to perform the operation as two bytes. A halfword aligned word load or store adds one extra cycle to perform the operation as two halfwords. A byte-aligned word load or store adds two extra cycles to perform the operation as a byte, a halfword, and a byte. These numbers increase if the memory stalls."

However, multi-word memory instructions (LDRD, STRD, LDM, STM, etc.) always require their arguments to be word-aligned.

Re: Redis on the Raspberry Pi: Adventures in unaligned lands

#47

Post by throwaway000002 »

pm215 wrote: post #43
throwaway000002 wrote: post #41

Earlier quoted context omitted.

I can understand the historical requirements for alignment, the necessary transistors, what not. But, much like branch-delay slots, there is no modern reason to expose this to the programmer. Of course, I gave an exception to atomics, but if you will, they're like memory-mapped communication, and now that all I/O is memory-mapped, with no concept of ports, the (ordering) semantics of memory access becomes really impo…

Even today, allowing unaligned accesses is still not free -- there is an implementation cost in transistors and in design complexity. There's a tradeoff here, as usual. There are a lot of places with a CPU architecture where there's a choice of "do we handle this in hardware, at the cost of having to have more hardware, or do we say it's software's job to deal with this, and hardware provides either nothing or just s…

You sound like an architecture person (I'm not, btw), so maybe you can give the lowdown on this.

Why registers? I haven't studied the Tomasulo algorithm in any detail, but if you're going to do "register renaming", why have registers at all? You could, for example, treat memory as a if-needed-only backing store, and then add a "commit" instruction that commits memory (takes an address, or a range). Sure you need to make changes with how you do mm i/o and protection, but at a basic level: why registers?

I'm glad FPGA's are becoming a thing, and I think we're about a decade or two away from ASICs as a service, because if you're not beholden to tradition, you really can work some magic. Of course I'll be pretty rusty by then, but who knows, maybe medicine will keep me feisty.

Re: Redis on the Raspberry Pi: Adventures in unaligned lands

#48

Post by tedunangst »

throwaway000002 wrote: post #35
wzdd wrote: post #33

Earlier quoted context omitted.

> if you support byte-addressing you'd better as well be happy with byte-alignment All ARM processors do this. The concept is called "natural alignment" and it's pretty common on non-x86. See e.g. http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.... . The problem here is that a lot of code written for x86 wants more than that, e.g. byte addressing for non-byte-wide values.

I understand. What I mean is that if your word-size is not your addressing-size, you'd better not have a concept of mis-aligned accesses. It's trouble you brought on all by yourself.

The cray did this, iirc, and the result was that char pointers were extra fat because they needed to include the word address and the byte address within the word. That's not an efficiency improvement.

Re: Redis on the Raspberry Pi: Adventures in unaligned lands

#49

Post by yeswecatan »

luhn wrote: post #36

> Redis is adding a “Stream” data type that is specifically suited for streams of data and time series storage, at this point the specification is near complete and work to implement it will start in the next weeks. This sounds like it could be really exciting. Is there anywhere I can find out more? Specifically, I've been struggling to find an appropriate backend for HTTP Server-Sent Events, could this feature help…

Here's a discussion on reddit. There's a link to the proposal on github, too.

https://www.reddit.com/r/redis/comments/4mmrgr/stream_data_s...

Post reply on HN