> Consider, for instance, bitcasting a [2]u8 to a u16. Under the old semantics, the result of this operation depends on the target endian: on big-endian targets, the first array element became the 8 most significant bits, whereas on little-endian targets, the first array element became the 8 least significant bits. Under the new semantics, because we only care about logical bit representation (which is endian-agnostic), the operation behaves identically on every target:
This is a huge mistake. You would never expect something like bitCast to do this.
I don't understand this approach. Why change something so simple and low level to be complicated and high level?
Just don't allow casting to u24, as it makes no sense unless you define u24 to be u32 sized as I think c standard does.
I think this approach as an idea is bad but at least just add another built-in that implements this higher level idea to not break a simple expectation and current behavior?
If I understand it correctly, it basically boils down to copying bits from the source to the destination, in order from the least significant bit to the most significant bit. It's not equivalent to C++'s reinterpret_cast.
I'm no Zig expert, but if you want endian-dependent semantics I'd assume either @ptrCast or a packed union would do the job.
> This is a huge mistake. You would never expect something like bitCast to do this.
Is there at least some sort of @transmute or something ? If Zig wants to say "bitCast" means this odd operation, but provides the thing most people actually want under some plausible name that's just an extra thing to learn which seems OK.
This change + the existing packed struct logic will be great for working with bit packed binary headers w/o having to manually twiddle so much about the bit handling along the way.
Interesting read, even as someone who isn't using Zig.
I wonder, these arbitrary-width integers... Is it actually even really worth it? My intuition is to prefer manually packing/unpacking things instead (in any language, even C that has bit width for struct fields), because it gives me a better mental picture of the code that is actually generated. Particularly for something like an signed odd-bit integer - what kind of code gets generated for sign-extension, a presumably common operation?
Does anybody have other experiences with them, one way or the other?
IIRC, for "normal" bit widths the codegen basically uses the next larger machine type and preserves zero bits on the high end. An i3 is an i8 with five MSB zeroes (with more custom behavior for "packed" i3 values). It's UB to fill those with non-zero values. For larger bit widths, like u729, you concatenate many large machine types, the compiler generates instructions in an unrolled loop, and the LLVM optimization pass usually doesn't clean that up (though, now that integers are apparently not using the LLVM u729 implementation, perhaps there are some more optimization opportunities).
They're situationally useful, especially when performance isn't an enormous concern. That u729 example above came from a variant sudoku solver I wrote to aid developing new puzzles (easy to check the rough magnitude of the solution space for whatever idea I was mulling over and examine how restricted the board actually was -- just an intermediate step in puzzle design). It's not optimal (hard on the icache, can be hard on registers, other issues abound), but it's dead simple to use, and the assembly isn't terrible, beating all the normal solvers I saw floating around. It's a nice point on the laziness/correctness/good-enough-perf pareto curve.
Another comment mentioned this, but they're great in packed structs for representing weird numeric entities (I think I have a logarithmic number system floating around which does that).
One thing the language does quite a lot is use them to guard against certain classes of human error at compile time. It doesn't perfectly make impossible actions unrepresentable, but shoving a full u32 into a shift argument usually doesn't make sense, so the types are constrained to be smaller.
I can't imagine any situation where I'd use a u729 instead of a StaticBitSet. For size 729, it would end up backed by a bit_set.Array, not a bit_set.Integer.
I don't program zig, so it's not clear to me if you can use zig's bitsets arithmetically.
Sometimes it's just more clear to work with integers than other representations. Most situations with a state space of N bits have meaningful integer representations, where arithmetic functions on those representations are also meaningful.
For example, CRCs can be written as the remainder from long division of the message by the polynomial. Defining nontrivial cyclic permutations is also much more straightforward as functions on integers than on bitsets.
If I had to steel-man the idea, I'm pretty sure the integer-based solution has better codegen with many kinds of sparse, comptime-known masks. I think you're right though, StaticBitSet looks better.
As an fpga engineer dealing with bitwidths that are non-byte multiples is very normal and when I end up writing software for various reasons, I often miss it. Usually when trying to slice and parse or construct messages.
Obviously there are ways around pretty much everything, but it’s nice to have first class language support for bit slices.
except it isn't bit slice, it isn't indexing within a range - it's just integer type that only allows values up to 2^width, with same alignment rounding up as with the rest
It's a bit slice if you put it in a packed struct.
I like them, they're nicer than C's bitfields: The order isn't implementation-defined, and the types remember their range rather than being converted to a power-of-two size upon read. (Maybe that's possible with C23 _BitInt(n), I haven't tried if those work in bitfields)
> Quite long devlog coming up, apologies—I got a little carried away with this one!
mlugg, please don't apologize for creating something I actually want to read. I'm drowning in low effort garbage, the in depth technical explanation is a refreshing breath of fresh air.
Might as well apologize for creating a language without a garbage collector, sure most people are unwilling to think, but some of us like nice things and are actually willing to apply effort.
OT: I'm always surprised at how popular Zig discussions get here, or Youtube and other medias.
Don't get me wrong, I love Zig and I think it's a great C replacement, but I'm very confused on why C3 or Odin rarely get any attention at all, despite being in the same C-replacement crowd.
But still surprised at what Zig does better than these other projects? Is Andrew much better at marketing/promoting the language? He's very hard to dislike.
I think Andrew is a big part of it, and the people he surrounded himself with are the other part.
What kind of pre-1.0 language hosts conventions? Crazy that they manage to do that.
Andrew's vision has always been clear and inspiring to me. I think this got Zig its initial following, and they have capitalized extremely well on it to grow as a community.
Andrew doesn't strike me as someone who does any marketing at all. He just wants to make the language he wants to use, and does it well.
Sometimes its just right time, right place. But also, Zig has received attention via projects like Ghostty, TigerBeetle, and Bun (prior to rewrite of course)
> Consider, for instance, bitcasting a [2]u8 to a u16. Under the old semantics, the result of this operation depends on the target endian: on big-endian targets, the first array element became the 8 most significant bits, whereas on little-endian targets, the first array element became the 8 least significant bits. Under the new semantics, because we only care about logical bit representation (which is endian-agnostic), the operation behaves identically on every target:
This is a huge mistake. You would never expect something like bitCast to do this.
I don't understand this approach. Why change something so simple and low level to be complicated and high level?
Just don't allow casting to u24, as it makes no sense unless you define u24 to be u32 sized as I think c standard does.
I think this approach as an idea is bad but at least just add another built-in that implements this higher level idea to not break a simple expectation and current behavior?
If I understand it correctly, it basically boils down to copying bits from the source to the destination, in order from the least significant bit to the most significant bit. It's not equivalent to C++'s reinterpret_cast.
I'm no Zig expert, but if you want endian-dependent semantics I'd assume either @ptrCast or a packed union would do the job.
You don't need to use @bitCast for the behavior you're talking about. @ptrCast still exists.
> This is a huge mistake. You would never expect something like bitCast to do this.
Is there at least some sort of @transmute or something ? If Zig wants to say "bitCast" means this odd operation, but provides the thing most people actually want under some plausible name that's just an extra thing to learn which seems OK.
This change + the existing packed struct logic will be great for working with bit packed binary headers w/o having to manually twiddle so much about the bit handling along the way.
Interesting read, even as someone who isn't using Zig.
I wonder, these arbitrary-width integers... Is it actually even really worth it? My intuition is to prefer manually packing/unpacking things instead (in any language, even C that has bit width for struct fields), because it gives me a better mental picture of the code that is actually generated. Particularly for something like an signed odd-bit integer - what kind of code gets generated for sign-extension, a presumably common operation?
Does anybody have other experiences with them, one way or the other?
IIRC, for "normal" bit widths the codegen basically uses the next larger machine type and preserves zero bits on the high end. An i3 is an i8 with five MSB zeroes (with more custom behavior for "packed" i3 values). It's UB to fill those with non-zero values. For larger bit widths, like u729, you concatenate many large machine types, the compiler generates instructions in an unrolled loop, and the LLVM optimization pass usually doesn't clean that up (though, now that integers are apparently not using the LLVM u729 implementation, perhaps there are some more optimization opportunities).
They're situationally useful, especially when performance isn't an enormous concern. That u729 example above came from a variant sudoku solver I wrote to aid developing new puzzles (easy to check the rough magnitude of the solution space for whatever idea I was mulling over and examine how restricted the board actually was -- just an intermediate step in puzzle design). It's not optimal (hard on the icache, can be hard on registers, other issues abound), but it's dead simple to use, and the assembly isn't terrible, beating all the normal solvers I saw floating around. It's a nice point on the laziness/correctness/good-enough-perf pareto curve.
Another comment mentioned this, but they're great in packed structs for representing weird numeric entities (I think I have a logarithmic number system floating around which does that).
One thing the language does quite a lot is use them to guard against certain classes of human error at compile time. It doesn't perfectly make impossible actions unrepresentable, but shoving a full u32 into a shift argument usually doesn't make sense, so the types are constrained to be smaller.
I can't imagine any situation where I'd use a u729 instead of a StaticBitSet. For size 729, it would end up backed by a bit_set.Array, not a bit_set.Integer.
https://ziglang.org/documentation/master/std/#std.bit_set.St...
I don't program zig, so it's not clear to me if you can use zig's bitsets arithmetically.
Sometimes it's just more clear to work with integers than other representations. Most situations with a state space of N bits have meaningful integer representations, where arithmetic functions on those representations are also meaningful.
For example, CRCs can be written as the remainder from long division of the message by the polynomial. Defining nontrivial cyclic permutations is also much more straightforward as functions on integers than on bitsets.
Old habits :)
If I had to steel-man the idea, I'm pretty sure the integer-based solution has better codegen with many kinds of sparse, comptime-known masks. I think you're right though, StaticBitSet looks better.
It's great for defining fancy floats used in machine learning
e.g. https://github.com/zml/zml/blob/33ced8fa078b3c7c8c709bd526ae...
As an fpga engineer dealing with bitwidths that are non-byte multiples is very normal and when I end up writing software for various reasons, I often miss it. Usually when trying to slice and parse or construct messages.
Obviously there are ways around pretty much everything, but it’s nice to have first class language support for bit slices.
except it isn't bit slice, it isn't indexing within a range - it's just integer type that only allows values up to 2^width, with same alignment rounding up as with the rest
It's a bit slice if you put it in a packed struct.
I like them, they're nicer than C's bitfields: The order isn't implementation-defined, and the types remember their range rather than being converted to a power-of-two size upon read. (Maybe that's possible with C23 _BitInt(n), I haven't tried if those work in bitfields)
> Quite long devlog coming up, apologies—I got a little carried away with this one!
mlugg, please don't apologize for creating something I actually want to read. I'm drowning in low effort garbage, the in depth technical explanation is a refreshing breath of fresh air.
Might as well apologize for creating a language without a garbage collector, sure most people are unwilling to think, but some of us like nice things and are actually willing to apply effort.
I appreciate the kind words :)
It wasn't even long! It seemed much shorter than the typical LLM-expanded drivel that crosses the HN front page daily.
> sure most people are unwilling to think, but some of us like nice things and are actually willing to apply effort.
Sir, you seem to have dropped your fedora.
??
OT: I'm always surprised at how popular Zig discussions get here, or Youtube and other medias.
Don't get me wrong, I love Zig and I think it's a great C replacement, but I'm very confused on why C3 or Odin rarely get any attention at all, despite being in the same C-replacement crowd.
But still surprised at what Zig does better than these other projects? Is Andrew much better at marketing/promoting the language? He's very hard to dislike.
I think Andrew is a big part of it, and the people he surrounded himself with are the other part. What kind of pre-1.0 language hosts conventions? Crazy that they manage to do that. Andrew's vision has always been clear and inspiring to me. I think this got Zig its initial following, and they have capitalized extremely well on it to grow as a community.
Andrew doesn't strike me as someone who does any marketing at all. He just wants to make the language he wants to use, and does it well.
Sometimes its just right time, right place. But also, Zig has received attention via projects like Ghostty, TigerBeetle, and Bun (prior to rewrite of course)