04:50 |
cgra |
signpost, in ocpy Decoder.solve(), wouldn't "destination_block_buffer ^= block.buffer" be better as "destination_block_buffer[:] = block.buffer" ? |
04:53 |
cgra |
first operand in a series of xor ops not needed to xor with initial zero |
| |
~ 3 hours 8 minutes ~ |
08:01 |
cgra |
signpost, did you ever consider if aux blocks could be generated on-the-fly, instead of pre-generated set? (i thought this option was mentioned in the paper(s), but can't find it anymore, or i was just imagining it) |
08:09 |
cgra |
appears to me you cannot get exact same aux block probability distribution if you want to map from an aux block to 0..n message blocks, instead of mapping each message block to q aux blocks, but perhaps don't need exact same distribution? |
08:10 |
signpost |
will need moar coffee to answer this question, but you're not supposed to use the same prng for aux blocks and check blocks, so when you generate aux blocks shouldn't affect that, I'd think. |
08:10 |
signpost |
check blocks all get seeded with their block ID |
08:10 |
signpost |
*production of check-block edges gets seeded with ^ |
08:11 |
signpost |
(also thanks for reading the thing! glad my thick skull isn't the only one aimed at it!) |
08:11 |
signpost |
ty phf also |
| |
~ 33 minutes ~ |
08:44 |
cgra |
signpost, yeah i didn't mean to mix aux and check block prng's, but was pondering whether by slightly(?) changing the aux rng usage could you avoid mandatory aux pre-gen and generate aux blocks (still deterministically) as they are required |
08:46 |
signpost |
aux blocks are a lot fatter than check blocks, and there's only say 3% of them compared to check blocks. I'd think doing lazy aux block generation would introduce chug in encoding each time one's hit as a check block edge |
08:46 |
signpost |
that said there's no reason not to do the experiment. |
08:47 |
signpost |
you're supposed to be able to run the second-phase of encoding indefinitely, while first-phase that produces the aux blocks is finite and small-ish, so I just ran it up front. |
08:48 |
signpost |
one thing you don't necessarily want to leak to an outside adversary is which blocks rely upon aux blocks, because that'd give targeting info to the adversary. and if all aux-block-yielding check blocks are nuked, you're not getting the payload on the other side |
08:48 |
signpost |
encoding's guarantees rely upon the enemy not being able to discern which blocks to zap |
08:49 |
cgra |
signpost, don't pre-genned aux blocks already give such info? because don't need any genning |
08:50 |
signpost |
I don't think so, because to the check block pass they're just more message blocks |
08:50 |
signpost |
there's not more or less work in the check-block phase because something was an aux block vs message block. it's just another edge that gets xor'd into check block payload |
08:51 |
cgra |
ah right |
08:51 |
signpost |
vs if every time a check block edge ended up pointing at an aux block we then generated the aux block (which has many more edges than most check blocks) there'd be discernible choog |
08:52 |
cgra |
yes, true |
08:57 |
cgra |
could you say that on-the-fly aux-blocks only giving operator a ~choice~ whether to pre-gen/cache or not, makes the timing leak a minor issue? |
09:02 |
cgra |
signpost, np re reading ocpy, and double the ty for creating it! |
09:06 |
signpost |
as I think about it, the timing leak is both an issue there and in one other case. in the case we're discussing, I'd rather not introduce configurability that persuades folks to do something which makes it easier for an attacker to undermine the reason for the encoding in the first place. |
| |
↖ |
09:06 |
signpost |
and it occurs to me that this timing issue applies to all block generation. |
09:07 |
signpost |
if the sender *ever* exposes their rate of block generation by letting blocks go out faster than they can be generated, an attacker can infer by timing the degree of all check blocks. |
09:08 |
signpost |
at which point it's as easy as zapping degree-one check blocks, which ensure nothing ever gets solved. |
09:08 |
signpost |
degree-1 blocks having one edge, and being identical to the underlying message or aux block. |
09:08 |
cgra |
signpost, i suppose on-the-fly auxes beneficial also to pre-gen by not requiring to process all aux blocks in one unit |
09:08 |
signpost |
solving a degree-n block is impossible when total solved blocks is not at least the number of that block's degree, right? |
| |
↖ |
09:09 |
signpost |
so if you don't get any degree-1 blocks... "we ain't found shit!" |
09:09 |
signpost |
https://www.youtube.com/watch?v=g3iFJpGJiug |
09:10 |
cgra |
http://logs.bitdash.io/pest/2023-02-03#1022094 << makes sense |
09:10 |
bitbot |
Logged on 2023-02-03 09:06:37 signpost: as I think about it, the timing leak is both an issue there and in one other case. in the case we're discussing, I'd rather not introduce configurability that persuades folks to do something which makes it easier for an attacker to undermine the reason for the encoding in the first place. |
09:14 |
cgra |
http://logs.bitdash.io/pest/2023-02-03#1022100 << is that off-by-one? otherwise agreed |
09:14 |
bitbot |
Logged on 2023-02-03 09:08:48 signpost: solving a degree-n block is impossible when total solved blocks is not at least the number of that block's degree, right? |
09:14 |
signpost |
herp, yep |
09:14 |
signpost |
at least degree-minus-one blocks have to be solved for it to be possible (but not guaranteed) that a check block can be solved |
09:15 |
* |
signpost sips moar coffee |
09:23 |
signpost |
hm. I'm having a hard time seeing how this can be solved without reducing send rate to the worst encoding case, throwing random delays or random dummy blocks in, etc. |
09:23 |
signpost |
curious what folks think. |
09:25 |
signpost |
via radio, would be harder to zap particular blocks, but via internet, appears not. |
09:26 |
jonsykkel |
if enemy can zap at all, they can just zap all ur bloks no |
09:28 |
signpost |
depends whether we're modeling DDoS or packet-analysis/silent-dropping |
09:28 |
signpost |
but yeah, that's true |
09:29 |
cgra |
signpost, say you randomize it. wouldn't some slip through eventually? |
09:29 |
signpost |
if DDoS, the encoding can be tuned to make use of whatever loss rate, can be 99.999%. if there's any signal getting through, reassembly of the message will be possible over some amount of time. |
09:30 |
signpost |
cgra: randomizing, keeping a check-block hopper full and limiting transmission to a rate which ensures hopper doesn't run dry, all those kinds of mitigations. |
09:30 |
cgra |
right |
09:31 |
jonsykkel |
yep, i mean specificaly re the picky zapping. guess they can make it 3% less obvious that zapping is going on |
09:33 |
signpost |
just worthwhile to evaluate whether this encoding does anything, or whether it'd be better to just piss file-chunks as fast as possible, and maaaaybe have this thing available as the tool it's designed for, to handle squeeking out information when under heavy loss. |
09:34 |
* |
signpost happy to say the thing doesn't do what's needed if it doesn't. |
09:35 |
signpost |
another thing that oughta be modeled out is the swarm download aspect. the fact that separate nodes can create compatible streams of a given file makes for easy, uncoordinated collaboration to get another node a file. |
09:35 |
* |
signpost will think moar and return in a bit |
| |
~ 1 hours 10 minutes ~ |
10:46 |
* |
asciilifeform not grasps how the 'enemy knows which packets to drop' problem could be relevant outside of a compromised key |
10:46 |
signpost |
yeah, I guess you'd have to know some luby-farts are going on at all, before deciding to drop the easiest to generate |
10:47 |
signpost |
but I'd think the lubystreams would look different from the outside. constant packet spray, variable time between packets |
10:47 |
asciilifeform |
snoop does not know which blacket packets contain what, tho |
10:47 |
asciilifeform |
even if can make good guess somehow that packet p is part of a luby stream -- has nfi what part |
10:47 |
signpost |
maybe not a problem, just figured it oughta be pushed on from this side, see if stands. |
10:48 |
signpost |
the "what part" would be inferrable via timing if packets leave the station as soon as they're encoded, right? |
10:49 |
signpost |
the encoded packets have a linear variability in difficulty to produce. another xor-encoding edge from encoded block to underlying message block would add same amount of encoding time. |
10:49 |
asciilifeform |
aint the ~whole point of lubyism to make order (as well as loss of some arbitrary %) not matter ? |
10:50 |
signpost |
*if and only if the enemy cannot discern packets. |
10:50 |
signpost |
and I'm pushing on whether the timing leak makes sufficiently discernibl |
10:50 |
signpost |
*le |
10:50 |
asciilifeform |
so buffer'em and send at fixed rate (which you gotta do anyway if width of receiving and transmitting pipes differ) |
10:50 |
signpost |
yup, was mentioned as a solution |
10:51 |
* |
asciilifeform still catching up on log |
10:51 |
signpost |
if that's satisfactory to you too, sounds good. |
10:51 |
signpost |
yep np |
10:52 |
asciilifeform |
aha. buffer in such way that a burst containing mix of 'easy' and 'hard' chunks is pre-genned and then sent |
10:57 |
asciilifeform |
hypothetically not even need buffer, if can generate 'easy' and 'hard' chunks in parallel, avoiding 'interesting' idle intervals on the line |
10:58 |
asciilifeform |
then all you gotta do is match the rate to the receiver's |
10:58 |
bitbot |
Logged on 2022-12-29 12:34:01 phf[awt]: here, catch! |
10:58 |
signpost |
ah yeah, the more parallelism you have going in encoding the better off you are on this front, as well as the obvious one of having more throughput. |
11:02 |
signpost |
ok, pretty satisfied this isn't a problem since we have to control the spray rate in any case. |
11:03 |
* |
signpost meanwhile seeing about chopping a simpler xorshift into the py version. will leave both in the code since phf has implemented the current one. not planning on making this configurable, but rather want to confirm that behavior on my end doens't change, or if so why. |
| |
↖ |
11:14 |
cgra |
signpost, if feel like it, there's also the 'un-floating' to consider simultaneously, plus the on-the-fly aux is related |
11:15 |
signpost |
could you restate the benefit of on the fly aux generation? |
11:19 |
cgra |
1) smaller memory requirement: processing one aux block at a time vs all aux blocks at once, 2) no mandatory extra state, 'just files' and the encoder |
11:19 |
signpost |
yeah but would have to reprocess these aux blocks every time referenced, or cache |
11:20 |
cgra |
yes |
11:20 |
signpost |
oughta be measured maybe, and see the hit to different payload sizes |
11:21 |
cgra |
also uncertain to my feeble brain if can 'invert' the probability math for it |
11:22 |
signpost |
another solution to "what if the payload's large, and we're talking about 30G of aux blocks" is to chunk streams into finite encoding scopes. |
11:22 |
signpost |
doing that might give what's needed to estimate how fast the other end can eat. |
11:23 |
signpost |
and establishes points where adjustments can be made to encoding parameters. |
11:23 |
cgra |
would poor man's version of ^ be splitting bluray images into .part1,.part2 etc? |
11:24 |
signpost |
yeah, encoder would chunk the payload like that. |
11:25 |
signpost |
say receiver stops receiving blocks, and hasn't solved a chunk. can signal back "hey, I'm at 300/305 blocks solved over here on chunk 5" |
11:25 |
signpost |
then sender can compare that to his previous estimate and adjust because loss was 5 blocks. |
11:25 |
signpost |
receiver can also politely signal back that they received more blocks than they needed, and sender can adjust down. |
11:26 |
signpost |
or sender can periodically decay and see what happens. |
11:33 |
signpost |
were I to generate the aux blocks on the fly, next decision would be whether to lazily advance the prng which produces the message-aux edges, or precompute that. |
| |
↖ |
11:34 |
signpost |
feels like fine tuning, not a core issue. |
11:40 |
phf |
http://logs.bitdash.io/pest/2023-02-03#1022148 << xorshift1024 fails LinearComp on "Big Crush tests" per https://arxiv.org/pdf/1810.05313.pdf, i don't know what that means for us |
| |
↖ |
11:40 |
bitbot |
Logged on 2023-02-03 11:03:37 signpost: meanwhile seeing about chopping a simpler xorshift into the py version. will leave both in the code since phf has implemented the current one. not planning on making this configurable, but rather want to confirm that behavior on my end doens't change, or if so why. |
11:41 |
phf |
but then there's another prng quality analysis page, where it's not listed as failing linearcomp, https://prng.di.unimi.it |
11:41 |
signpost |
not sure. fwiw I've got a test in the pywad that drops packets and confirms that the message can still be solved. |
11:42 |
phf |
yah, i don't understand oc at all yet, so i'm just leaving this here |
11:44 |
phf |
http://logs.bitdash.io/pest/2023-02-02#1022068 << i figured out why it generates garbage on the last multiply. (unsigned-byte 64) is not immediate on a 64-bit lisp, on account of there not being enough space to put a tag. so to not make it generate garbage it needs to be inlined |
11:44 |
bitbot |
Logged on 2023-02-02 17:19:57 phf[deedbot|awt]: e.g. xorsfhit1024 becomes http://paste.deedbot.org/?id=UZma, and it still generates garbage on that last multiply (it bignums and then truncates, i can't figure out how to make it use a int64bit version of *) |
11:45 |
phf |
otherwise the value is boxed on return |
11:48 |
phf |
niether here nor there problem. to make things be easily fast on a lisp ought to keep bit size under most-positive-fixnum, which is short two bits under sbcl from whatever system word size |
11:48 |
cgra |
signpost, pre-gen/on-the-fly prolly otherwise a minor point, but affects aux randomization algo, so must choose which one to use |
11:49 |
phf |
cgra, i don't understand, why would on-the-fly require a different aux randomization algo? |
11:50 |
cgra |
phf, well, current is from a message block at a time, to aux blocks, but on-the-fly is 'this one here aux, which message blocks?' |
11:52 |
cgra |
i mean, the algo part around prng |
11:54 |
phf |
that didn't clarify, but i'll keep the point in mind while reading the code again |
11:59 |
cgra |
i'll try rephrasing sometime soon, atm bit busy off-desk |
| |
↖ |
12:07 |
asciilifeform |
http://logs.bitdash.io/pest/2023-02-03#1022167 << afaik non-ideal prng, in context of lubyism, simply gives less-than-ideal 'mileage' (excluding the case of a catastrophically-broken prng, e.g. with laughably short period; there, could fail entirely to send $block) |
12:07 |
bitbot |
Logged on 2023-02-03 11:40:23 phf[awt|deedbot]: http://logs.bitdash.io/pest/2023-02-03#1022148 << xorshift1024 fails LinearComp on "Big Crush tests" per https://arxiv.org/pdf/1810.05313.pdf, i don't know what that means for us |
12:08 |
signpost |
yeah, the channel would seem lossier the worse the rng. |
12:09 |
asciilifeform |
aaha |
12:09 |
signpost |
this property made for hilarious pitfalls throughout the implementation, where brain-dead bugs would still eventually work |
12:11 |
asciilifeform |
biased prng will simply send subset of chunks moar often than they oughta be sent, at the expense of some other subset |
12:15 |
phf |
yeah, but that's not what linear complexity is, hence the question of how lincomp affects us |
12:18 |
phf |
as in, we can just go and choose an algo that doesn't fail linearcomp, but there might be no reason to. |
12:19 |
asciilifeform |
phf: as asciilifeform understands, from pov of lubyism (rather than cryptographic context) all statistical failures of prng 'are same thing' |
12:19 |
phf |
yeah, well, that's wrong |
12:20 |
asciilifeform |
phf: possibly; but then plox to elaborate |
12:21 |
phf |
e.g. https://www.turingfinance.com/hacking-the-random-walk-hypothesis/#linear-complexity-test suggests that linear complexity test is a test by insinuation. it attempts to automatically construct a lsfr out of bit stream |
| |
↖ |
12:22 |
phf |
*for the bit stream rather |
12:23 |
phf |
so it approximates one from a bit stream, and then compares it's length against "what would be expected from a true uniform random sequence" |
12:23 |
asciilifeform |
so, for instance, does 'prng fails linear complexity test' imply that one can construct a luby payload that will never properly send (i.e. some blox systematically avoided) ? or 1 that takes 10x the expected # of chunks to send..? |
| |
↖ |
12:24 |
phf |
no, it doesn't even live in the same realm |
12:25 |
phf |
it's explicitly a xorshift detector (xorshift IS an lsfr), and if it detects that the sequence could've been produced with a short xorshift it says "that's awfully short, it's probably not random" |
12:25 |
signpost |
there's something head-cracking in there, at least for a dummy like me. |
12:26 |
signpost |
first, why xorshift is so good at this, for as few steps as it contains. |
12:26 |
signpost |
second, that apparently a lot of prng behavior is reducible into something like xorshift |
12:26 |
* |
signpost would like to understand the math better. |
12:39 |
phf |
asciilifeform, it's a kind of walking in circles :) my understanding so far is that lct is a "short" lsfr detector, but the main appeal of xorshift family is that it's a short lsfr. i don't understand (nor have i found laymen documentation) on what other implications this have |
12:46 |
phf |
i'll just leave this here as is, |
12:46 |
phf |
”These scrambled Xorshift generators output 64-bit integers. Ideally, a 64-bit PRNG should be tested with a 64-bit test suite; using a test suite designed for 32-bit PRNGs (or actually 31-bit PRNGs in the case of Big Crush) necessarily involves compomises. Vigna’s approach was |
12:46 |
phf |
to interleave the least-significant 32 bits and most-significant 32 bits of each 64-bit output, thus each 64-bit output becomes two 32-bit outputs. This interleaving strategy may detect flaws in some 64-bit PRNGs, but it may also hide statistical weaknesses. Such an approach is use |
12:46 |
phf |
ful but is insufficient. A more comprehensive strategy is to also test the most-significant and least-significant 32 bits separately. When we focus on the least-significant 32 bits, and, specifically, in the case where their bits are reversed, we find that the scrambled Xorshift ge |
12:46 |
phf |
nerators systematically fail Big Crush.” |
12:48 |
phf |
and then later |
12:48 |
phf |
“The generators under consid- eration require relatively long seeds (at least 16 bytes). To generate a sufficiently long seed (128 or 1024 bits), we produce a random integer with the Bash shell’s internal RANDOM function as an initial seed for to the 64-bit SplitMix gen- erator [8].” |
12:54 |
* |
asciilifeform went to read phf's 'random walks' link but head not came back yet, may take a while |
| |
~ 38 minutes ~ |
13:33 |
phf |
http://logs.bitdash.io/pest/2023-02-03#1022196 << from reading luby ideal selection is not random, but follows some kind of fancy distribution. i imagine oc is similar |
13:33 |
bitbot |
Logged on 2023-02-03 12:23:22 asciilifeform[4]: so, for instance, does 'prng fails linear complexity test' imply that one can construct a luby payload that will never properly send (i.e. some blox systematically avoided) ? or 1 that takes 10x the expected # of chunks to send..? |
13:33 |
phf |
not ideal, but best case |
| |
~ 15 minutes ~ |
13:48 |
phf |
http://logs.bitdash.io/pest/2023-02-03#1022193 << also nist test code ftr https://raw.githubusercontent.com/GINARTeam/NIST-statistical-test/master/10_linear_complexity_test.py |
13:48 |
bitbot |
Logged on 2023-02-03 12:21:31 phf[deedbot|awt]: e.g. https://www.turingfinance.com/hacking-the-random-walk-hypothesis/#linear-complexity-test suggests that linear complexity test is a test by insinuation. it attempts to automatically construct a lsfr out of bit stream |
13:48 |
asciilifeform |
phf: aha. was trying to think of 'what happens to it under variously sad prng' |
13:52 |
asciilifeform |
( given that transmitted chunks are reorderable, potentially, using an actual trng -- 'enemy may infer order' failure mode can be ruled out ) |
13:52 |
asciilifeform |
so afaik an attack would have to consist of a constructed payload that eats surprising bw, or even not completes |
13:53 |
asciilifeform |
( e.g. a hypothetical lolcat.jpg that loox unremarkable and may end up distributed on pest nets but somehow takes 1tb of lubyistic chunks to actually move ) |
13:56 |
asciilifeform |
rather like a de bruijn sequence worx against von neumann's fortune teller |
13:56 |
phf |
well, there's all kinds of questions as far as "malevolent counterparty" |
13:57 |
asciilifeform |
well on pestnet, properly malevolent counterparty can spam at will until unpeered. moar interesting imho q is re subtly boobytrapped payloads from third parties |
13:57 |
asciilifeform |
would seem possible if prng has a 'usable' flaw |
13:58 |
phf |
oc can be done with an explicit switch on and an explicit switched off, so e.g. can ask counterparty to send you lolcat then never ask to stop |
13:59 |
phf |
because only reciever know if fully recieved "oh, keep sending, keep sending, i still don't quite have the whole thing" can be an entirely legitimate state |
13:59 |
asciilifeform |
for instance. which is wai may need at least 'soft' peer bw quota, e.g. 'hey asciilifeform , why do you eat 99% of my bw asking for lolcat_part1_100.jpg 24/7 ?' |
13:59 |
asciilifeform |
need to at least indicate to operator that it is happening, imho |
14:00 |
asciilifeform |
given peer oughn't be able to soak up station bw 24/7 silently |
14:00 |
phf |
right, or some probabilistic checks "ok, dude, you should have had the thing by now unless your channel have 0.9999 error rate" |
14:01 |
asciilifeform |
even nao a broken or mischevious peer can throw getdata at you repeatedly |
14:01 |
asciilifeform |
(if you dun ration resources, it'll thrash disk and slow yer station to a crawl) |
14:02 |
phf |
yeah, but that's to your point of "one for one", luby explicitly fails the one for one |
14:03 |
asciilifeform |
correct, tho with getdata, 'one for one' is only tru re bw, given as sending getdata is considerably cheaper than answering it |
14:03 |
asciilifeform |
(if results in disk seek) |
14:03 |
phf |
rright |
14:04 |
phf |
also you can definitely do ddos with luby. as of right no handshake in protocol, so you can e.g. forge a valid "ignore" packet, but with bogus origin ip, so that will update AT, and then you follow it up by "turn on luby please for lolcat.jpg" |
14:05 |
asciilifeform |
( hence wai per spec, getdata is only serviced if 'cpu cycles available' ) |
14:05 |
phf |
*naive luby that is |
14:06 |
phf |
or better yet "turn on luby for all the files that i know you have" |
14:07 |
asciilifeform |
if you've the key, can indeed trigger lubyism to arbitrary ip, if can route forged packet to receiver |
14:10 |
asciilifeform |
fwiw this needs not only peer key, but also to know of a valid file hash to make luby req for |
14:12 |
asciilifeform |
i.e. joker can try to use his l1 as a 'ddos cannon' |
| |
↖ |
14:12 |
phf |
right |
14:13 |
asciilifeform |
imho the Right Thing to do is to make such lulz visible to operators, rather than try to rule out mechanically |
14:14 |
asciilifeform |
'hey asciilifeform , what do you think yer doing?' |
14:15 |
phf |
there can be a legitimate reason for doing something like that, for example that would be direct effect of initial log sync |
| |
↖ |
14:15 |
asciilifeform |
the other obv pill would be to require a handshake before firing off luby sequence |
| |
↖ |
14:15 |
asciilifeform |
at least 1 packet each way |
14:15 |
phf |
yah |
14:15 |
asciilifeform |
sorta like the handshake used in 'rekey' |
14:16 |
phf |
http://logs.bitdash.io/pest/2023-02-03#1022247 << your initial sync is a bandwidth heavy cornucopia. long stream of getdata requests, and depending on how you implement, a getdata for a message with luby payload results in "oh yeah and by the way also give me stream for this luby" |
14:16 |
bitbot |
Logged on 2023-02-03 14:15:00 phf[awt|deedbot]: there can be a legitimate reason for doing something like that, for example that would be direct effect of initial log sync |
14:17 |
phf |
in fact i can see how a naive log sync might result in a self-ddos :D |
14:17 |
asciilifeform |
well per current spec, only text msgs are getdata'ble |
14:17 |
asciilifeform |
lubyism in asciilifeform's conception is entirely separate protocol, with transport via pest directmsgs |
14:17 |
phf |
yes, but yes, but log will eventually contain hashes for payload, as per discussion |
14:17 |
asciilifeform |
aha |
14:18 |
asciilifeform |
doesn't mean one necessarily has to fetch erry single one during log sync tho |
14:18 |
phf |
but it would be a legitimate use case |
14:18 |
asciilifeform |
sure. you'll end up with over9000 linux isos etc if you do it, tho |
14:18 |
asciilifeform |
typically not what you want |
14:19 |
phf |
that's probably not true, on account of y'all punting on the "essential" payload question |
14:20 |
phf |
if you're catching up with log, it's sensible then to have some kind of heuristic for automatic luby payload request, "give me all png/jpg/gif/pdf files that are less than 20mb" or whatever |
14:20 |
* |
asciilifeform not expects to find a satisfying mechanical answer to that q |
14:21 |
asciilifeform |
in practice, will prolly end up as e.g. 'gimme erry bin signpost offered up' |
14:21 |
phf |
well, that too |
14:22 |
phf |
but for example whatever my client can display inline (images, video, audio), i'd want to fetch as soon as it's available |
14:22 |
asciilifeform |
how to not turn 'hey l1, gimme luby of lolcat.jpg' into a guaranteed own-goal is genuinely open q (suppose yer l1 is 20 stations and they all have it, say) atm |
14:23 |
asciilifeform |
sorta wai asciilifeform not convinced that it oughta default to 'hey l1...' but rather start with 1, and request others as-needed |
14:25 |
asciilifeform |
elementarily, if you're on e.g. 1g/s pipe, you dun win anyffin from having 20 1g/s pipes spewing at you at line rate simult. |
14:26 |
asciilifeform |
if yer isp is typical konsoomer sack of sad, may even trigger a ddos sensor somewhere and getcha dropped a la pizarro and you'll be picking up phone to complain 'to central laundry' |
14:29 |
phf |
but isn't that the mechanical beauty of luby. you get 20 streams, you get the result 20x faster :> |
14:29 |
phf |
right, but the requirement that you should be able to negotiate pipe rate was heavily discussed because there are other reasons |
14:29 |
phf |
don't want to get a single line rate stream from a 1 20g/s pipe either |
14:29 |
phf |
bump |
14:29 |
phf |
kek, internet went down so a handful of messages didn't go out. on reconnect i asked it to bump http://glyf.org/screenshots/pest-bump.png |
14:29 |
phf |
n stations sending same luby stream is part of the same probability space |
14:30 |
asciilifeform |
lol until saturated line (supposing you even want the thing badly enuff that you dun mind saturating) |
14:31 |
phf |
asciilifeform, yeah but you ask n peers for maxrate/n luby streams |
14:31 |
asciilifeform |
phf's station seems to be pioneering testbed for 'can operate on poor-quality net pipe' design goal |
14:31 |
asciilifeform |
phf: right, can do this if you know for fact that the folx yer asking in fact all have lolcat.jpg |
14:31 |
phf |
it's entirely roaming too. like i just close the laptop and open it at cafe, it self recovers (by spewing ignore packets to everyone, once it can) |
14:31 |
asciilifeform |
(and that all have cycles to spare, and will send, etc) |
14:32 |
phf |
asciilifeform, so you have to have a provision to renegotiate. in fact "send lolcat at 10" followed by "send lolcat at 100" should produce correct rate adjustment result |
14:32 |
asciilifeform |
phf: is precisely how is supposed to work (still need the 'hammer' thing, for nat, tho) |
14:33 |
asciilifeform |
asciilifeform's notion is that one would request subhashes, e.g. 1MB at a time |
14:33 |
phf |
why then luby at all? |
14:33 |
asciilifeform |
(if payload smaller than MB, there are no subs) |
14:33 |
asciilifeform |
phf: balance between synchronicity and 'don't throw 1TB at me asynchronously plox' |
14:34 |
asciilifeform |
i.e. avoid the fairy tale magick pot that endlessly spews porridge and drowns the world |
14:34 |
phf |
my understanding is the whole point of fountain code is that you can ask bunch of random places to start sending you stuff, and with a high probability you're going to full your block holes in a reasonable amount of time |
14:34 |
asciilifeform |
limiting subchunk to e.g. MB lets ya say 'pot, stop' |
14:34 |
cgra |
what's wrong with 1MB of check blocks per 'luby getdata'? |
14:35 |
asciilifeform |
right, but imho not unreasonable to require that the process stops after certain # of blox, unless requested again |
14:36 |
asciilifeform |
otherwise yer in a position where e.g. asciilifeform asks for lolcat.jpg, then loses net connection and never says 'enuff', and nao yer sending 4evah (or until some rather large timeout) |
14:37 |
phf |
that should not be timeout based |
14:37 |
cgra |
say, 'luby getdata' = '1MB more plz' |
14:37 |
* |
asciilifeform not knows ideal cutoff, but does not think it oughta be much larger than 1MB |
14:38 |
asciilifeform |
the point of making exception from 'want a packet? send 1 packet' is to economize outgoing bw. but ought not to be at the cost of potentially drowning yerself errytime |
14:39 |
asciilifeform |
(as well as wasting over9000 of errybody's outgoing bw by having'em piss at you over9000 moar packets than actually need) |
14:41 |
asciilifeform |
another angle to think of it from, is that the total avail. bw to all the participants in a net is a finite resource, and ought not be burned carelessly |
14:41 |
phf |
i think some kind of hard limit on luby stream makes sense, so like getme lolcat 10, will send you a 1mb of lolcat blocks at 10b/s or whatever. if that was not enough you can ask again and again etc. |
14:42 |
asciilifeform |
aaha |
14:42 |
phf |
but the idea of "gimme a specific chunk but with luby" ought to be tested against the inherent properties of luby |
14:43 |
* |
asciilifeform still suspects that best heuristic is '1 peer may not ever account for > 1/n bw (where n is # of peers) except by explicit permission of station operator' |
14:43 |
asciilifeform |
indeed oughta be tested. |
14:43 |
dulapbot |
Logged on 2023-01-29 13:43:03 asciilifeform: ... and many of these details, imho, can only be nailed down empirically. |
14:47 |
cgra |
why specific chunks again? to review partial data before full 60GB bluray image complete? |
14:47 |
asciilifeform |
cgra: you get a tree of subhashes, and then know when you've actually received given portions of the payload |
14:48 |
asciilifeform |
if e.g. some peer is sending you rubbish, you know about it sooner |
14:48 |
asciilifeform |
( afaik none of the currently-known lubyistic encodings are at all resistant to rubbish ) |
14:48 |
cgra |
asciilifeform, right |
14:49 |
asciilifeform |
if sumbody's 'poisoning' yer warez, you want to find out asap. |
14:49 |
phf |
asciilifeform, but you can do that using crc sum on a block, rather than a whole new subrange mechanism |
14:49 |
cgra |
asciilifeform, seems like most check block solving starts when like ~90% already received |
14:50 |
cgra |
(or similar high%) |
14:50 |
asciilifeform |
(how? let's say, if some chunk seems to never converge, you start excluding packets from various peers and see whether then converges) |
14:51 |
asciilifeform |
phf: making the requestable unit of transmission a e.g. 1MB block with a known hash, is reasonably simple way to do it |
14:52 |
asciilifeform |
you request H. H is either a hash of entire payload (if <1MB) or a hash of H1+H2, which a cooperating peer will give you; H1 and H2 may in turn be hashes of H1_1, H1_2, H2_1, H2_2, and so forth |
14:52 |
asciilifeform |
until get to the leaves |
14:53 |
phf |
you're trying to bolt a tradition chunk synchronization mechanism on top of a very different model, and there are ways to achieve what you're tyring to do, but in a more luby way |
14:53 |
jonsykkel |
http://logs.bitdash.io/pest/2023-02-03#1022248 << if u can forge the ignore packet src ip to update AT, u can forge handshake ack in same way |
14:53 |
bitbot |
Logged on 2023-02-03 14:15:04 asciilifeform[5]: the other obv pill would be to require a handshake before firing off luby sequence |
14:53 |
asciilifeform |
possibly |
14:53 |
asciilifeform |
jonsykkel: to do handshake, you gotta be able to both send ~and receive~ from $ip |
14:53 |
asciilifeform |
i.e. it's 'yours' |
14:54 |
phf |
jonsykkel, "handshake" implicitly involves a shared secret |
| |
↖ |
14:54 |
jonsykkel |
ah ok i see |
14:54 |
asciilifeform |
(err, both receive at and send from) |
14:54 |
asciilifeform |
and yes |
14:55 |
asciilifeform |
yet anuther reason wai you want to discretize subchunks -- say you switched cafes mid-download |
14:55 |
asciilifeform |
once yer peers run outta the given 1MB, they'll send the next one to yer new AT |
14:56 |
asciilifeform |
(given as that's where you'll be asking for the next MB from) |
14:57 |
asciilifeform |
(and you'll be re-requesting the interrupted subchunk, obv) |
14:59 |
asciilifeform |
the main reason you gotta, afaik, have the hash tree, tho, is to readily detect poisoning. |
15:00 |
asciilifeform |
otherwise you can sit there and, having received over9000 chunks, be left wondering wai never converges. |
15:02 |
asciilifeform |
whereas if yer expecting to converge on a 1MB chunk of known hash, and sumbody's already send you over9000x the nominally-req'd luby chunks, and still not converges -- you'll know that 1 or more of the suppliers is fucking you |
15:03 |
asciilifeform |
nao, the ~clever~ way to cut this knot would be if you could somehow nail down, via hash tree, the expected hashes of individual luby chunks |
15:03 |
asciilifeform |
rather than arbitrary cut into e.g. 1MB |
15:04 |
asciilifeform |
there'll still be a discrete subchunk size, tho, much larger than indiv. packet, tho |
15:04 |
asciilifeform |
(or tree itself would have to be impractically large) |
15:04 |
asciilifeform |
really oughta determine empirically. |
15:04 |
* |
asciilifeform must bbl |
15:09 |
cgra |
phf, trying this again: |
15:09 |
bitbot |
Logged on 2023-02-03 11:59:43 cgra[jonsykkel]: i'll try rephrasing sometime soon, atm bit busy off-desk |
15:11 |
cgra |
current randomization of edges between message blocks and aux blocks is by drawing exactly 3 aux blocks to each message block. |
15:14 |
cgra |
but if coming from on-the-fly angle, you'd have to first draw 'how many message blocks for this aux block', perhaps using the aux block itself as a seed of sorts, and after that draw as many message blocks |
15:17 |
cgra |
in theory, a single aux block can have 0..<message block count> edges while every message block has exactly 3 edges (with current ocpy params) |
15:21 |
cgra |
in above on-the-fly model, ocpy Header.message_block_edges and Header.aux_block_edges don't exist |
15:26 |
cgra |
(single aux block edges theoretical max 3 * <message block count> with current ocpy params, made mistake above) |
15:34 |
cgra |
a check block edge count ranges from 1 to 2114, averaging at ~8. if doing correct math here, aux block edge count averages at ~182 |
15:34 |
cgra |
again ^ per ocpy current params |
15:37 |
cgra |
if you have a 100GB my-dumbuntu.zip, you have ~800 million message block/aux block edges |
15:40 |
cgra |
and ~4 million aux blocks |
| |
~ 1 hours 37 minutes ~ |
17:17 |
asciilifeform |
http://logs.bitdash.io/pest/2023-02-03#1022328 << to elaborate: all you need is, before sending e.g. MB of luby, sent to the proposed addr 'here's 256 random bytes, plox to send'em back, then i'll start' |
17:17 |
bitbot |
Logged on 2023-02-03 14:54:04 phf[awt|deedbot]: jonsykkel, "handshake" implicitly involves a shared secret |
17:17 |
asciilifeform |
*send to |
17:18 |
asciilifeform |
the luby gets sent strictly to the addr which sent'em back, no AT renewal permitted between the 2 msgs |
17:23 |
jonsykkel |
indeed |
17:26 |
phf |
should that be some kind of outside of luby mechanism? |
17:27 |
phf |
like you come online, start sending ignores, station recognizes change in at, and send you a challenge, that you need to respond to |
| |
~ 43 minutes ~ |
18:10 |
phf |
actually disregard, that seems like a bad idea, since it introduces all kinds of potential issues on the core messaging side |
18:11 |
phf |
one for one is such a nice property, once you violate it for reasons, suddenly need all kinds of extra mechanisms. |
| |
~ 27 minutes ~ |
18:39 |
asciilifeform |
phf: the 'send me back these rng bytes' thing would be part of the trigger for the luby (so 'outside', yes). imho would be enuff to avoid 'cannon' w/out adding much gnarl |
18:39 |
bitbot |
Logged on 2023-02-03 14:12:05 asciilifeform[jonsykkel]: i.e. joker can try to use his l1 as a 'ddos cannon' |
18:41 |
asciilifeform |
concretely, instead of a single binary direct msg 'gimme lubystream of hash H' being answered immed. by stream, the station will answer 'sure, but 1st send me back these rng bytes, stream will go to ip they get send back from and it has to be the same 1 from which you had asked just nao' |
18:44 |
asciilifeform |
no need to recognize, as such, change in AT; simply insist that the addr to which luby piss is to go is able to receive, as well as send, under the requester's key |
18:45 |
asciilifeform |
you can't forge, lol, ~receiving~ (or rather it aint called 'forging' anymoar if you can..) |
18:50 |
asciilifeform |
is very similar to the handshake spec already has for rekey, the purpose of which is to prevent a buggy, sad-rng'd, or saboteur peer from forcing the peering key to an apriori-known value |
18:53 |
asciilifeform |
( there's naturally no way to prevent an idjit or traitor from divulging key to third party. but he can't force it to e.g. zero ) |
18:58 |
asciilifeform |
... this may be the 1 and only original idea, lol, in pest protocol. (tho for all asciilifeform knows, even it existed somewhere. but nfi where) |
18:58 |
asciilifeform |
contrast with e.g. ssl's 'null cipher'(tm) magick bit. |
| |
~ 1 hours 14 minutes ~ |
20:12 |
phf |
right, i was over engineering things in my head. simply "guarding" luby with a handshake is good enough |
20:14 |
phf |
in totally unrelated can someone explain the china weather balloon story to me? i think ~that~ finally broke my clown world processing machinery |
| |
~ 35 minutes ~ |
20:49 |
crtdaydreams |
^ second that, nfi where "spy balloon" nonsense origniated, but from a cursory glance anyone with a modicum of intelligence would observe that a) why worry about the bitch on the porch when there's a bear in the livingroom? and b) why not satellite imagery? lmao |
20:52 |
* |
crtdaydreams recalls a public satellite imagery website where could view .hk images and naviage around, could zoom in far enough to make out peoples faces quite easily |
20:54 |
crtdaydreams |
to the extent that faces etc. would be higher quality than found on e.g. passport |
20:55 |
phf |
that's not the angle i'm even interested in, more like "why is this such a big deal" and "why is there no sane grown up reaction" |
20:56 |
phf |
1950s solution, announce that it's a weather balloon, immediately mount a televised recovery operation, have spooks look over it on a tight schedule, have a pr transfer of the balloon the chinese event, "our slanty eyed best partners seem to have lost their toy, we give it back, to |
20:56 |
phf |
bright future!" everyone drinks champagne and smiles |
20:56 |
phf |
2023 zoomer solution "ugh awkward ugh we're monitoring *shuffles awkwardly around*" |
20:56 |
signpost |
yeah, the lack of theatric message-sending is bizarre. |
20:56 |
phf |
w t f |
20:57 |
signpost |
phf: I saw earlier that chinese state media of some kind was laughing at us being unable to control our airspace, but failed to find it a second time |
20:58 |
asciilifeform |
balloons aint obsolete, 'cheap & angry' if conspicuous snoop camera |
20:58 |
asciilifeform |
Official excuse for not popping it, apparently: 'could cause property damage on the ground' lol |
20:58 |
signpost |
curious if it's scanning radio |
20:59 |
phf |
the lack of any kind of sane grown up reaction is bizarre. i mean predictable, but that's like the ultimate failure of "hello yes hello" sanity check on there being present any kind of intelligence in the decision making aparatus |
20:59 |
signpost |
wifi, bluetooth, etc for exploitation |
| |
↖ |
20:59 |
signpost |
maybe they want to dump it in the ocean and recover |
20:59 |
asciilifeform |
hard to imagine wai else to bake such a large one if not to carry luxurious antennae |
20:59 |
crtdaydreams |
can't knock it down because "Das Racist!!1111!" |
21:00 |
crtdaydreams |
lolol |
21:01 |
phf |
signpost, in an obviously public event like that one ought to signal an intent, whatever the strategy is |
21:01 |
signpost |
sadder answer would be that biden's crackhead son relayed a threat from the chicoms. |
21:01 |
* |
asciilifeform pictures that there's likely a usg 'golden toilet' wunderwaffe designed to steal balloons, but not actually worx, and officials scheming to blame one anuther |
21:01 |
signpost |
and then he made up a reason to order no action. |
21:02 |
signpost |
mil could be internally wtf-ing at grandpa's order. |
21:02 |
signpost |
not claiming certainty, but any answer suggests woeful unpreparedness for war. |
21:06 |
crtdaydreams |
too busy waging war on own denizens |
21:08 |
phf |
so kommuniti consensus is wwiii? |
21:10 |
crtdaydreams |
phf: doubtful imo |
21:10 |
phf |
somehow in my world view chinese lost a balloon, "weather balloon" for diplomatic purposes, would've been mildly embarrassed but u.s. saved them face by being even more awkward and incompetent at diplomacy |
21:13 |
signpost |
chatter says the thing flew over a lot of air force bases. |
21:13 |
signpost |
and that it was maneuvering. |
21:14 |
signpost |
https://www.youtube.com/watch?v=g3iFJpGJiug << e.g. |
21:14 |
signpost |
whiteman's a nuke base, b-2s etc |
21:14 |
signpost |
I think they scanned our MSFT-running trash. |
21:17 |
phf |
is it being piloted by one hide ping sam, the infamous ghost of beijing? |
21:19 |
signpost |
poor guy's instructed to jump out in the atlantic and start drinking. |
21:25 |
phf |
antenas are for show, hide ping sam remembers every detail of enemy installations by heart |
| |
~ 17 minutes ~ |
21:42 |
crtdaydreams |
lol |
| |
~ 38 minutes ~ |
22:21 |
asciilifeform |
http://logs.bitdash.io/pest/2023-02-03#1022390 << bluetooth/wifi from 24e3 km is a bit lul imho |
22:21 |
bitbot |
Logged on 2023-02-03 20:59:16 signpost: wifi, bluetooth, etc for exploitation |
22:21 |
asciilifeform |
err, 24km |
22:34 |
signpost |
lol, I somehow pasted the spaceballs thing again. |
22:34 |
signpost |
link was meant to be a senator twisting his nipples about whiteman base |
22:35 |
signpost |
ironic name, at that. |