I first encountered q/kdb+ at a quant job in 2007. I learned so much from the array semantics about how to concisely represent time-series logic that I can't imagine ever using a scalar language for research.
Fun fact: the aj (asof join) function was my inspiration for pandas.merge_asof. I added the extra parameters (direction, tolerance, allow_exact_matches) because of the limitations I kept hitting in kdb.
The aj function at its heart is a bin (https://code.kx.com/q/ref/bin/) search between the two tables, on the requested columns, to find the indices of the right table to zip onto the left table.
aj[`sym`time;t;q]
becomes
t,'(`sym`time _q)(`sym`time#q)bin`sym`time#t
The rest of the aj function internals are there to handle edge cases, handling missing columns and options for filling nulls.
A lot of the joins can be distilled to the core operators/functions in a similar manner. For example the plus-join is
I couldn't figure-out how Arthur's bin matched on symbol though, so I switched to a linear scan on the right table to record the last-seen index for each "by" element. While it worked, my hash table was messy because I relied on Python to handle a whole tuple as a key, which had some issues during initial testing.
The asof join I wrote for Empirical properly categorizes the keys before they are matched. That approach worked far better.
I first encountered q/kdb+ at a quant job in 2007. I learned so much from the array semantics about how to concisely represent time-series logic that I can't imagine ever using a scalar language for research.
Fun fact: the aj (asof join) function was my inspiration for pandas.merge_asof. I added the extra parameters (direction, tolerance, allow_exact_matches) because of the limitations I kept hitting in kdb.
https://code.kx.com/q/ref/aj/
https://pandas.pydata.org/docs/reference/api/pandas.merge_as...
The aj function at its heart is a bin (https://code.kx.com/q/ref/bin/) search between the two tables, on the requested columns, to find the indices of the right table to zip onto the left table.
becomes The rest of the aj function internals are there to handle edge cases, handling missing columns and options for filling nulls.A lot of the joins can be distilled to the core operators/functions in a similar manner. For example the plus-join is
Indeed, my very first attempt used numpy.searchsorted:
https://numpy.org/doc/2.2/reference/generated/numpy.searchso...
I couldn't figure-out how Arthur's bin matched on symbol though, so I switched to a linear scan on the right table to record the last-seen index for each "by" element. While it worked, my hash table was messy because I relied on Python to handle a whole tuple as a key, which had some issues during initial testing.
The asof join I wrote for Empirical properly categorizes the keys before they are matched. That approach worked far better.
https://www.empirical-soft.com/tutorial.html#dataframes
Similarly, this is how it was introduced in ClickHouse in 2019: https://github.com/ClickHouse/ClickHouse/pull/4774
I wish there was a language like K that worked with single precision floats. Would be great to use with graphics.
q is like k and has single precision floats.
the github repo linked, under "manual.md", has a link which redirects to malware, such as that ostensibly to "k6".