Show Idle (>14 d.) Chans
← 2023-02-02 | 2023-02-04 → ↓
04:50 cgra signpost, in ocpy Decoder.solve(), wouldn't "destination_block_buffer ^= block.buffer" be better as "destination_block_buffer[:] = block.buffer" ?
04:53 cgra first operand in a series of xor ops not needed to xor with initial zero
~ 3 hours 8 minutes ~
08:01 cgra signpost, did you ever consider if aux blocks could be generated on-the-fly, instead of pre-generated set? (i thought this option was mentioned in the paper(s), but can't find it anymore, or i was just imagining it)
08:09 cgra appears to me you cannot get exact same aux block probability distribution if you want to map from an aux block to 0..n message blocks, instead of mapping each message block to q aux blocks, but perhaps don't need exact same distribution?
08:10 signpost will need moar coffee to answer this question, but you're not supposed to use the same prng for aux blocks and check blocks, so when you generate aux blocks shouldn't affect that, I'd think.
08:10 signpost check blocks all get seeded with their block ID
08:10 signpost *production of check-block edges gets seeded with ^
08:11 signpost (also thanks for reading the thing! glad my thick skull isn't the only one aimed at it!)
08:11 signpost ty phf also
~ 33 minutes ~
08:44 cgra signpost, yeah i didn't mean to mix aux and check block prng's, but was pondering whether by slightly(?) changing the aux rng usage could you avoid mandatory aux pre-gen and generate aux blocks (still deterministically) as they are required
08:46 signpost aux blocks are a lot fatter than check blocks, and there's only say 3% of them compared to check blocks. I'd think doing lazy aux block generation would introduce chug in encoding each time one's hit as a check block edge
08:46 signpost that said there's no reason not to do the experiment.
08:47 signpost you're supposed to be able to run the second-phase of encoding indefinitely, while first-phase that produces the aux blocks is finite and small-ish, so I just ran it up front.
08:48 signpost one thing you don't necessarily want to leak to an outside adversary is which blocks rely upon aux blocks, because that'd give targeting info to the adversary. and if all aux-block-yielding check blocks are nuked, you're not getting the payload on the other side
08:48 signpost encoding's guarantees rely upon the enemy not being able to discern which blocks to zap
08:49 cgra signpost, don't pre-genned aux blocks already give such info? because don't need any genning
08:50 signpost I don't think so, because to the check block pass they're just more message blocks
08:50 signpost there's not more or less work in the check-block phase because something was an aux block vs message block. it's just another edge that gets xor'd into check block payload
08:51 cgra ah right
08:51 signpost vs if every time a check block edge ended up pointing at an aux block we then generated the aux block (which has many more edges than most check blocks) there'd be discernible choog
08:52 cgra yes, true
08:57 cgra could you say that on-the-fly aux-blocks only giving operator a ~choice~ whether to pre-gen/cache or not, makes the timing leak a minor issue?
09:02 cgra signpost, np re reading ocpy, and double the ty for creating it!
09:06 signpost as I think about it, the timing leak is both an issue there and in one other case. in the case we're discussing, I'd rather not introduce configurability that persuades folks to do something which makes it easier for an attacker to undermine the reason for the encoding in the first place.
↖
09:06 signpost and it occurs to me that this timing issue applies to all block generation.
09:07 signpost if the sender *ever* exposes their rate of block generation by letting blocks go out faster than they can be generated, an attacker can infer by timing the degree of all check blocks.
09:08 signpost at which point it's as easy as zapping degree-one check blocks, which ensure nothing ever gets solved.
09:08 signpost degree-1 blocks having one edge, and being identical to the underlying message or aux block.
09:08 cgra signpost, i suppose on-the-fly auxes beneficial also to pre-gen by not requiring to process all aux blocks in one unit
09:08 signpost solving a degree-n block is impossible when total solved blocks is not at least the number of that block's degree, right?
↖
09:09 signpost so if you don't get any degree-1 blocks... "we ain't found shit!"
09:09 signpost https://www.youtube.com/watch?v=g3iFJpGJiug
09:10 cgra http://logs.bitdash.io/pest/2023-02-03#1022094 << makes sense
09:10 bitbot Logged on 2023-02-03 09:06:37 signpost: as I think about it, the timing leak is both an issue there and in one other case. in the case we're discussing, I'd rather not introduce configurability that persuades folks to do something which makes it easier for an attacker to undermine the reason for the encoding in the first place.
09:14 cgra http://logs.bitdash.io/pest/2023-02-03#1022100 << is that off-by-one? otherwise agreed
09:14 bitbot Logged on 2023-02-03 09:08:48 signpost: solving a degree-n block is impossible when total solved blocks is not at least the number of that block's degree, right?
09:14 signpost herp, yep
09:14 signpost at least degree-minus-one blocks have to be solved for it to be possible (but not guaranteed) that a check block can be solved
09:15 * signpost sips moar coffee
09:23 signpost hm. I'm having a hard time seeing how this can be solved without reducing send rate to the worst encoding case, throwing random delays or random dummy blocks in, etc.
09:23 signpost curious what folks think.
09:25 signpost via radio, would be harder to zap particular blocks, but via internet, appears not.
09:26 jonsykkel if enemy can zap at all, they can just zap all ur bloks no
09:28 signpost depends whether we're modeling DDoS or packet-analysis/silent-dropping
09:28 signpost but yeah, that's true
09:29 cgra signpost, say you randomize it. wouldn't some slip through eventually?
09:29 signpost if DDoS, the encoding can be tuned to make use of whatever loss rate, can be 99.999%. if there's any signal getting through, reassembly of the message will be possible over some amount of time.
09:30 signpost cgra: randomizing, keeping a check-block hopper full and limiting transmission to a rate which ensures hopper doesn't run dry, all those kinds of mitigations.
09:30 cgra right
09:31 jonsykkel yep, i mean specificaly re the picky zapping. guess they can make it 3% less obvious that zapping is going on
09:33 signpost just worthwhile to evaluate whether this encoding does anything, or whether it'd be better to just piss file-chunks as fast as possible, and maaaaybe have this thing available as the tool it's designed for, to handle squeeking out information when under heavy loss.
09:34 * signpost happy to say the thing doesn't do what's needed if it doesn't.
09:35 signpost another thing that oughta be modeled out is the swarm download aspect. the fact that separate nodes can create compatible streams of a given file makes for easy, uncoordinated collaboration to get another node a file.
09:35 * signpost will think moar and return in a bit
~ 1 hours 10 minutes ~
10:46 * asciilifeform not grasps how the 'enemy knows which packets to drop' problem could be relevant outside of a compromised key
10:46 signpost yeah, I guess you'd have to know some luby-farts are going on at all, before deciding to drop the easiest to generate
10:47 signpost but I'd think the lubystreams would look different from the outside. constant packet spray, variable time between packets
10:47 asciilifeform snoop does not know which blacket packets contain what, tho
10:47 asciilifeform even if can make good guess somehow that packet p is part of a luby stream -- has nfi what part
10:47 signpost maybe not a problem, just figured it oughta be pushed on from this side, see if stands.
10:48 signpost the "what part" would be inferrable via timing if packets leave the station as soon as they're encoded, right?
10:49 signpost the encoded packets have a linear variability in difficulty to produce. another xor-encoding edge from encoded block to underlying message block would add same amount of encoding time.
10:49 asciilifeform aint the ~whole point of lubyism to make order (as well as loss of some arbitrary %) not matter ?
10:50 signpost *if and only if the enemy cannot discern packets.
10:50 signpost and I'm pushing on whether the timing leak makes sufficiently discernibl
10:50 signpost *le
10:50 asciilifeform so buffer'em and send at fixed rate (which you gotta do anyway if width of receiving and transmitting pipes differ)
10:50 signpost yup, was mentioned as a solution
10:51 * asciilifeform still catching up on log
10:51 signpost if that's satisfactory to you too, sounds good.
10:51 signpost yep np
10:52 asciilifeform aha. buffer in such way that a burst containing mix of 'easy' and 'hard' chunks is pre-genned and then sent
10:57 asciilifeform hypothetically not even need buffer, if can generate 'easy' and 'hard' chunks in parallel, avoiding 'interesting' idle intervals on the line
10:58 asciilifeform then all you gotta do is match the rate to the receiver's
10:58 bitbot Logged on 2022-12-29 12:34:01 phf[awt]: here, catch!
10:58 signpost ah yeah, the more parallelism you have going in encoding the better off you are on this front, as well as the obvious one of having more throughput.
11:02 signpost ok, pretty satisfied this isn't a problem since we have to control the spray rate in any case.
11:03 * signpost meanwhile seeing about chopping a simpler xorshift into the py version. will leave both in the code since phf has implemented the current one. not planning on making this configurable, but rather want to confirm that behavior on my end doens't change, or if so why.
↖
11:14 cgra signpost, if feel like it, there's also the 'un-floating' to consider simultaneously, plus the on-the-fly aux is related
11:15 signpost could you restate the benefit of on the fly aux generation?
11:19 cgra 1) smaller memory requirement: processing one aux block at a time vs all aux blocks at once, 2) no mandatory extra state, 'just files' and the encoder
11:19 signpost yeah but would have to reprocess these aux blocks every time referenced, or cache
11:20 cgra yes
11:20 signpost oughta be measured maybe, and see the hit to different payload sizes
11:21 cgra also uncertain to my feeble brain if can 'invert' the probability math for it
11:22 signpost another solution to "what if the payload's large, and we're talking about 30G of aux blocks" is to chunk streams into finite encoding scopes.
11:22 signpost doing that might give what's needed to estimate how fast the other end can eat.
11:23 signpost and establishes points where adjustments can be made to encoding parameters.
11:23 cgra would poor man's version of ^ be splitting bluray images into .part1,.part2 etc?
11:24 signpost yeah, encoder would chunk the payload like that.
11:25 signpost say receiver stops receiving blocks, and hasn't solved a chunk. can signal back "hey, I'm at 300/305 blocks solved over here on chunk 5"
11:25 signpost then sender can compare that to his previous estimate and adjust because loss was 5 blocks.
11:25 signpost receiver can also politely signal back that they received more blocks than they needed, and sender can adjust down.
11:26 signpost or sender can periodically decay and see what happens.
11:33 signpost were I to generate the aux blocks on the fly, next decision would be whether to lazily advance the prng which produces the message-aux edges, or precompute that.
↖
11:34 signpost feels like fine tuning, not a core issue.
11:40 phf http://logs.bitdash.io/pest/2023-02-03#1022148 << xorshift1024 fails LinearComp on "Big Crush tests" per https://arxiv.org/pdf/1810.05313.pdf, i don't know what that means for us
↖
11:40 bitbot Logged on 2023-02-03 11:03:37 signpost: meanwhile seeing about chopping a simpler xorshift into the py version. will leave both in the code since phf has implemented the current one. not planning on making this configurable, but rather want to confirm that behavior on my end doens't change, or if so why.
11:41 phf but then there's another prng quality analysis page, where it's not listed as failing linearcomp, https://prng.di.unimi.it
11:41 signpost not sure. fwiw I've got a test in the pywad that drops packets and confirms that the message can still be solved.
11:42 phf yah, i don't understand oc at all yet, so i'm just leaving this here
11:44 phf http://logs.bitdash.io/pest/2023-02-02#1022068 << i figured out why it generates garbage on the last multiply. (unsigned-byte 64) is not immediate on a 64-bit lisp, on account of there not being enough space to put a tag. so to not make it generate garbage it needs to be inlined
11:44 bitbot Logged on 2023-02-02 17:19:57 phf[deedbot|awt]: e.g. xorsfhit1024 becomes http://paste.deedbot.org/?id=UZma, and it still generates garbage on that last multiply (it bignums and then truncates, i can't figure out how to make it use a int64bit version of *)
11:45 phf otherwise the value is boxed on return
11:48 phf niether here nor there problem. to make things be easily fast on a lisp ought to keep bit size under most-positive-fixnum, which is short two bits under sbcl from whatever system word size
11:48 cgra signpost, pre-gen/on-the-fly prolly otherwise a minor point, but affects aux randomization algo, so must choose which one to use
11:49 phf cgra, i don't understand, why would on-the-fly require a different aux randomization algo?
11:50 cgra phf, well, current is from a message block at a time, to aux blocks, but on-the-fly is 'this one here aux, which message blocks?'
11:52 cgra i mean, the algo part around prng
11:54 phf that didn't clarify, but i'll keep the point in mind while reading the code again
11:59 cgra i'll try rephrasing sometime soon, atm bit busy off-desk
↖
12:07 asciilifeform http://logs.bitdash.io/pest/2023-02-03#1022167 << afaik non-ideal prng, in context of lubyism, simply gives less-than-ideal 'mileage' (excluding the case of a catastrophically-broken prng, e.g. with laughably short period; there, could fail entirely to send $block)
12:07 bitbot Logged on 2023-02-03 11:40:23 phf[awt|deedbot]: http://logs.bitdash.io/pest/2023-02-03#1022148 << xorshift1024 fails LinearComp on "Big Crush tests" per https://arxiv.org/pdf/1810.05313.pdf, i don't know what that means for us
12:08 signpost yeah, the channel would seem lossier the worse the rng.
12:09 asciilifeform aaha
12:09 signpost this property made for hilarious pitfalls throughout the implementation, where brain-dead bugs would still eventually work
12:11 asciilifeform biased prng will simply send subset of chunks moar often than they oughta be sent, at the expense of some other subset
12:15 phf yeah, but that's not what linear complexity is, hence the question of how lincomp affects us
12:18 phf as in, we can just go and choose an algo that doesn't fail linearcomp, but there might be no reason to.
12:19 asciilifeform phf: as asciilifeform understands, from pov of lubyism (rather than cryptographic context) all statistical failures of prng 'are same thing'
12:19 phf yeah, well, that's wrong
12:20 asciilifeform phf: possibly; but then plox to elaborate
12:21 phf e.g. https://www.turingfinance.com/hacking-the-random-walk-hypothesis/#linear-complexity-test suggests that linear complexity test is a test by insinuation. it attempts to automatically construct a lsfr out of bit stream
↖
12:22 phf *for the bit stream rather
12:23 phf so it approximates one from a bit stream, and then compares it's length against "what would be expected from a true uniform random sequence"
12:23 asciilifeform so, for instance, does 'prng fails linear complexity test' imply that one can construct a luby payload that will never properly send (i.e. some blox systematically avoided) ? or 1 that takes 10x the expected # of chunks to send..?
↖
12:24 phf no, it doesn't even live in the same realm
12:25 phf it's explicitly a xorshift detector (xorshift IS an lsfr), and if it detects that the sequence could've been produced with a short xorshift it says "that's awfully short, it's probably not random"
12:25 signpost there's something head-cracking in there, at least for a dummy like me.
12:26 signpost first, why xorshift is so good at this, for as few steps as it contains.
12:26 signpost second, that apparently a lot of prng behavior is reducible into something like xorshift
12:26 * signpost would like to understand the math better.
12:39 phf asciilifeform, it's a kind of walking in circles :) my understanding so far is that lct is a "short" lsfr detector, but the main appeal of xorshift family is that it's a short lsfr. i don't understand (nor have i found laymen documentation) on what other implications this have
12:46 phf i'll just leave this here as is,
12:46 phf 􏿽”These scrambled Xorshift generators output 64-bit integers. Ideally, a 64-bit PRNG should be tested with a 64-bit test suite; using a test suite designed for 32-bit PRNGs (or actually 31-bit PRNGs in the case of Big Crush) necessarily involves compomises. Vigna’s approach was
12:46 phf 􏿽to interleave the least-significant 32 bits and most-significant 32 bits of each 64-bit output, thus each 64-bit output becomes two 32-bit outputs. This interleaving strategy may detect flaws in some 64-bit PRNGs, but it may also hide statistical weaknesses. Such an approach is use
12:46 phf 􏿽ful but is insufficient. A more comprehensive strategy is to also test the most-significant and least-significant 32 bits separately. When we focus on the least-significant 32 bits, and, specifically, in the case where their bits are reversed, we find that the scrambled Xorshift ge
12:46 phf 􏿽nerators systematically fail Big Crush.”
12:48 phf and then later
12:48 phf “The generators under consid- eration require relatively long seeds (at least 16 bytes). To generate a sufficiently long seed (128 or 1024 bits), we produce a random integer with the Bash shell’s internal RANDOM function as an initial seed for to the 64-bit SplitMix gen- erator [8].”
12:54 * asciilifeform went to read phf's 'random walks' link but head not came back yet, may take a while
~ 38 minutes ~
13:33 phf http://logs.bitdash.io/pest/2023-02-03#1022196 << from reading luby ideal selection is not random, but follows some kind of fancy distribution. i imagine oc is similar
13:33 bitbot Logged on 2023-02-03 12:23:22 asciilifeform[4]: so, for instance, does 'prng fails linear complexity test' imply that one can construct a luby payload that will never properly send (i.e. some blox systematically avoided) ? or 1 that takes 10x the expected # of chunks to send..?
13:33 phf not ideal, but best case
~ 15 minutes ~
13:48 phf http://logs.bitdash.io/pest/2023-02-03#1022193 << also nist test code ftr https://raw.githubusercontent.com/GINARTeam/NIST-statistical-test/master/10_linear_complexity_test.py
13:48 bitbot Logged on 2023-02-03 12:21:31 phf[deedbot|awt]: e.g. https://www.turingfinance.com/hacking-the-random-walk-hypothesis/#linear-complexity-test suggests that linear complexity test is a test by insinuation. it attempts to automatically construct a lsfr out of bit stream
13:48 asciilifeform phf: aha. was trying to think of 'what happens to it under variously sad prng'
13:52 asciilifeform ( given that transmitted chunks are reorderable, potentially, using an actual trng -- 'enemy may infer order' failure mode can be ruled out )
13:52 asciilifeform so afaik an attack would have to consist of a constructed payload that eats surprising bw, or even not completes
13:53 asciilifeform ( e.g. a hypothetical lolcat.jpg that loox unremarkable and may end up distributed on pest nets but somehow takes 1tb of lubyistic chunks to actually move )
13:56 asciilifeform rather like a de bruijn sequence worx against von neumann's fortune teller
13:56 phf well, there's all kinds of questions as far as "malevolent counterparty"
13:57 asciilifeform well on pestnet, properly malevolent counterparty can spam at will until unpeered. moar interesting imho q is re subtly boobytrapped payloads from third parties
13:57 asciilifeform would seem possible if prng has a 'usable' flaw
13:58 phf oc can be done with an explicit switch on and an explicit switched off, so e.g. can ask counterparty to send you lolcat then never ask to stop
13:59 phf because only reciever know if fully recieved "oh, keep sending, keep sending, i still don't quite have the whole thing" can be an entirely legitimate state
13:59 asciilifeform for instance. which is wai may need at least 'soft' peer bw quota, e.g. 'hey asciilifeform , why do you eat 99% of my bw asking for lolcat_part1_100.jpg 24/7 ?'
13:59 asciilifeform need to at least indicate to operator that it is happening, imho
14:00 asciilifeform given peer oughn't be able to soak up station bw 24/7 silently
14:00 phf right, or some probabilistic checks "ok, dude, you should have had the thing by now unless your channel have 0.9999 error rate"
14:01 asciilifeform even nao a broken or mischevious peer can throw getdata at you repeatedly
14:01 asciilifeform (if you dun ration resources, it'll thrash disk and slow yer station to a crawl)
14:02 phf yeah, but that's to your point of "one for one", luby explicitly fails the one for one
14:03 asciilifeform correct, tho with getdata, 'one for one' is only tru re bw, given as sending getdata is considerably cheaper than answering it
14:03 asciilifeform (if results in disk seek)
14:03 phf rright
14:04 phf also you can definitely do ddos with luby. as of right no handshake in protocol, so you can e.g. forge a valid "ignore" packet, but with bogus origin ip, so that will update AT, and then you follow it up by "turn on luby please for lolcat.jpg"
14:05 asciilifeform ( hence wai per spec, getdata is only serviced if 'cpu cycles available' )
14:05 phf *naive luby that is
14:06 phf or better yet "turn on luby for all the files that i know you have"
14:07 asciilifeform if you've the key, can indeed trigger lubyism to arbitrary ip, if can route forged packet to receiver
14:10 asciilifeform fwiw this needs not only peer key, but also to know of a valid file hash to make luby req for
14:12 asciilifeform i.e. joker can try to use his l1 as a 'ddos cannon'
↖
14:12 phf right
14:13 asciilifeform imho the Right Thing to do is to make such lulz visible to operators, rather than try to rule out mechanically
14:14 asciilifeform 'hey asciilifeform , what do you think yer doing?'
14:15 phf there can be a legitimate reason for doing something like that, for example that would be direct effect of initial log sync
↖
14:15 asciilifeform the other obv pill would be to require a handshake before firing off luby sequence
↖
14:15 asciilifeform at least 1 packet each way
14:15 phf yah
14:15 asciilifeform sorta like the handshake used in 'rekey'
14:16 phf http://logs.bitdash.io/pest/2023-02-03#1022247 << your initial sync is a bandwidth heavy cornucopia. long stream of getdata requests, and depending on how you implement, a getdata for a message with luby payload results in "oh yeah and by the way also give me stream for this luby"
14:16 bitbot Logged on 2023-02-03 14:15:00 phf[awt|deedbot]: there can be a legitimate reason for doing something like that, for example that would be direct effect of initial log sync
14:17 phf in fact i can see how a naive log sync might result in a self-ddos :D
14:17 asciilifeform well per current spec, only text msgs are getdata'ble
14:17 asciilifeform lubyism in asciilifeform's conception is entirely separate protocol, with transport via pest directmsgs
14:17 phf yes, but yes, but log will eventually contain hashes for payload, as per discussion
14:17 asciilifeform aha
14:18 asciilifeform doesn't mean one necessarily has to fetch erry single one during log sync tho
14:18 phf but it would be a legitimate use case
14:18 asciilifeform sure. you'll end up with over9000 linux isos etc if you do it, tho
14:18 asciilifeform typically not what you want
14:19 phf that's probably not true, on account of y'all punting on the "essential" payload question
14:20 phf if you're catching up with log, it's sensible then to have some kind of heuristic for automatic luby payload request, "give me all png/jpg/gif/pdf files that are less than 20mb" or whatever
14:20 * asciilifeform not expects to find a satisfying mechanical answer to that q
14:21 asciilifeform in practice, will prolly end up as e.g. 'gimme erry bin signpost offered up'
14:21 phf well, that too
14:22 phf but for example whatever my client can display inline (images, video, audio), i'd want to fetch as soon as it's available
14:22 asciilifeform how to not turn 'hey l1, gimme luby of lolcat.jpg' into a guaranteed own-goal is genuinely open q (suppose yer l1 is 20 stations and they all have it, say) atm
14:23 asciilifeform sorta wai asciilifeform not convinced that it oughta default to 'hey l1...' but rather start with 1, and request others as-needed
14:25 asciilifeform elementarily, if you're on e.g. 1g/s pipe, you dun win anyffin from having 20 1g/s pipes spewing at you at line rate simult.
14:26 asciilifeform if yer isp is typical konsoomer sack of sad, may even trigger a ddos sensor somewhere and getcha dropped a la pizarro and you'll be picking up phone to complain 'to central laundry'
14:29 phf but isn't that the mechanical beauty of luby. you get 20 streams, you get the result 20x faster :>
14:29 phf right, but the requirement that you should be able to negotiate pipe rate was heavily discussed because there are other reasons
14:29 phf don't want to get a single line rate stream from a 1 20g/s pipe either
14:29 phf bump
14:29 phf kek, internet went down so a handful of messages didn't go out. on reconnect i asked it to bump http://glyf.org/screenshots/pest-bump.png
14:29 phf n stations sending same luby stream is part of the same probability space
14:30 asciilifeform lol until saturated line (supposing you even want the thing badly enuff that you dun mind saturating)
14:31 phf asciilifeform, yeah but you ask n peers for maxrate/n luby streams
14:31 asciilifeform phf's station seems to be pioneering testbed for 'can operate on poor-quality net pipe' design goal
14:31 asciilifeform phf: right, can do this if you know for fact that the folx yer asking in fact all have lolcat.jpg
14:31 phf it's entirely roaming too. like i just close the laptop and open it at cafe, it self recovers (by spewing ignore packets to everyone, once it can)
14:31 asciilifeform (and that all have cycles to spare, and will send, etc)
14:32 phf asciilifeform, so you have to have a provision to renegotiate. in fact "send lolcat at 10" followed by "send lolcat at 100" should produce correct rate adjustment result
14:32 asciilifeform phf: is precisely how is supposed to work (still need the 'hammer' thing, for nat, tho)
14:33 asciilifeform asciilifeform's notion is that one would request subhashes, e.g. 1MB at a time
14:33 phf why then luby at all?
14:33 asciilifeform (if payload smaller than MB, there are no subs)
14:33 asciilifeform phf: balance between synchronicity and 'don't throw 1TB at me asynchronously plox'
14:34 asciilifeform i.e. avoid the fairy tale magick pot that endlessly spews porridge and drowns the world
14:34 phf my understanding is the whole point of fountain code is that you can ask bunch of random places to start sending you stuff, and with a high probability you're going to full your block holes in a reasonable amount of time
14:34 asciilifeform limiting subchunk to e.g. MB lets ya say 'pot, stop'
14:34 cgra what's wrong with 1MB of check blocks per 'luby getdata'?
14:35 asciilifeform right, but imho not unreasonable to require that the process stops after certain # of blox, unless requested again
14:36 asciilifeform otherwise yer in a position where e.g. asciilifeform asks for lolcat.jpg, then loses net connection and never says 'enuff', and nao yer sending 4evah (or until some rather large timeout)
14:37 phf that should not be timeout based
14:37 cgra say, 'luby getdata' = '1MB more plz'
14:37 * asciilifeform not knows ideal cutoff, but does not think it oughta be much larger than 1MB
14:38 asciilifeform the point of making exception from 'want a packet? send 1 packet' is to economize outgoing bw. but ought not to be at the cost of potentially drowning yerself errytime
14:39 asciilifeform (as well as wasting over9000 of errybody's outgoing bw by having'em piss at you over9000 moar packets than actually need)
14:41 asciilifeform another angle to think of it from, is that the total avail. bw to all the participants in a net is a finite resource, and ought not be burned carelessly
14:41 phf i think some kind of hard limit on luby stream makes sense, so like getme lolcat 10, will send you a 1mb of lolcat blocks at 10b/s or whatever. if that was not enough you can ask again and again etc.
14:42 asciilifeform aaha
14:42 phf but the idea of "gimme a specific chunk but with luby" ought to be tested against the inherent properties of luby
14:43 * asciilifeform still suspects that best heuristic is '1 peer may not ever account for > 1/n bw (where n is # of peers) except by explicit permission of station operator'
14:43 asciilifeform indeed oughta be tested.
14:43 dulapbot Logged on 2023-01-29 13:43:03 asciilifeform: ... and many of these details, imho, can only be nailed down empirically.
14:47 cgra why specific chunks again? to review partial data before full 60GB bluray image complete?
14:47 asciilifeform cgra: you get a tree of subhashes, and then know when you've actually received given portions of the payload
14:48 asciilifeform if e.g. some peer is sending you rubbish, you know about it sooner
14:48 asciilifeform ( afaik none of the currently-known lubyistic encodings are at all resistant to rubbish )
14:48 cgra asciilifeform, right
14:49 asciilifeform if sumbody's 'poisoning' yer warez, you want to find out asap.
14:49 phf asciilifeform, but you can do that using crc sum on a block, rather than a whole new subrange mechanism
14:49 cgra asciilifeform, seems like most check block solving starts when like ~90% already received
14:50 cgra (or similar high%)
14:50 asciilifeform (how? let's say, if some chunk seems to never converge, you start excluding packets from various peers and see whether then converges)
14:51 asciilifeform phf: making the requestable unit of transmission a e.g. 1MB block with a known hash, is reasonably simple way to do it
14:52 asciilifeform you request H. H is either a hash of entire payload (if <1MB) or a hash of H1+H2, which a cooperating peer will give you; H1 and H2 may in turn be hashes of H1_1, H1_2, H2_1, H2_2, and so forth
14:52 asciilifeform until get to the leaves
14:53 phf you're trying to bolt a tradition chunk synchronization mechanism on top of a very different model, and there are ways to achieve what you're tyring to do, but in a more luby way
14:53 jonsykkel http://logs.bitdash.io/pest/2023-02-03#1022248 << if u can forge the ignore packet src ip to update AT, u can forge handshake ack in same way
14:53 bitbot Logged on 2023-02-03 14:15:04 asciilifeform[5]: the other obv pill would be to require a handshake before firing off luby sequence
14:53 asciilifeform possibly
14:53 asciilifeform jonsykkel: to do handshake, you gotta be able to both send ~and receive~ from $ip
14:53 asciilifeform i.e. it's 'yours'
14:54 phf jonsykkel, "handshake" implicitly involves a shared secret
↖
14:54 jonsykkel ah ok i see
14:54 asciilifeform (err, both receive at and send from)
14:54 asciilifeform and yes
14:55 asciilifeform yet anuther reason wai you want to discretize subchunks -- say you switched cafes mid-download
14:55 asciilifeform once yer peers run outta the given 1MB, they'll send the next one to yer new AT
14:56 asciilifeform (given as that's where you'll be asking for the next MB from)
14:57 asciilifeform (and you'll be re-requesting the interrupted subchunk, obv)
14:59 asciilifeform the main reason you gotta, afaik, have the hash tree, tho, is to readily detect poisoning.
15:00 asciilifeform otherwise you can sit there and, having received over9000 chunks, be left wondering wai never converges.
15:02 asciilifeform whereas if yer expecting to converge on a 1MB chunk of known hash, and sumbody's already send you over9000x the nominally-req'd luby chunks, and still not converges -- you'll know that 1 or more of the suppliers is fucking you
15:03 asciilifeform nao, the ~clever~ way to cut this knot would be if you could somehow nail down, via hash tree, the expected hashes of individual luby chunks
15:03 asciilifeform rather than arbitrary cut into e.g. 1MB
15:04 asciilifeform there'll still be a discrete subchunk size, tho, much larger than indiv. packet, tho
15:04 asciilifeform (or tree itself would have to be impractically large)
15:04 asciilifeform really oughta determine empirically.
15:04 * asciilifeform must bbl
15:09 cgra phf, trying this again:
15:09 bitbot Logged on 2023-02-03 11:59:43 cgra[jonsykkel]: i'll try rephrasing sometime soon, atm bit busy off-desk
15:11 cgra current randomization of edges between message blocks and aux blocks is by drawing exactly 3 aux blocks to each message block.
15:14 cgra but if coming from on-the-fly angle, you'd have to first draw 'how many message blocks for this aux block', perhaps using the aux block itself as a seed of sorts, and after that draw as many message blocks
15:17 cgra in theory, a single aux block can have 0..<message block count> edges while every message block has exactly 3 edges (with current ocpy params)
15:21 cgra in above on-the-fly model, ocpy Header.message_block_edges and Header.aux_block_edges don't exist
15:26 cgra (single aux block edges theoretical max 3 * <message block count> with current ocpy params, made mistake above)
15:34 cgra a check block edge count ranges from 1 to 2114, averaging at ~8. if doing correct math here, aux block edge count averages at ~182
15:34 cgra again ^ per ocpy current params
15:37 cgra if you have a 100GB my-dumbuntu.zip, you have ~800 million message block/aux block edges
15:40 cgra and ~4 million aux blocks
~ 1 hours 37 minutes ~
17:17 asciilifeform http://logs.bitdash.io/pest/2023-02-03#1022328 << to elaborate: all you need is, before sending e.g. MB of luby, sent to the proposed addr 'here's 256 random bytes, plox to send'em back, then i'll start'
17:17 bitbot Logged on 2023-02-03 14:54:04 phf[awt|deedbot]: jonsykkel, "handshake" implicitly involves a shared secret
17:17 asciilifeform *send to
17:18 asciilifeform the luby gets sent strictly to the addr which sent'em back, no AT renewal permitted between the 2 msgs
17:23 jonsykkel indeed
17:26 phf should that be some kind of outside of luby mechanism?
17:27 phf like you come online, start sending ignores, station recognizes change in at, and send you a challenge, that you need to respond to
~ 43 minutes ~
18:10 phf actually disregard, that seems like a bad idea, since it introduces all kinds of potential issues on the core messaging side
18:11 phf one for one is such a nice property, once you violate it for reasons, suddenly need all kinds of extra mechanisms.
~ 27 minutes ~
18:39 asciilifeform phf: the 'send me back these rng bytes' thing would be part of the trigger for the luby (so 'outside', yes). imho would be enuff to avoid 'cannon' w/out adding much gnarl
18:39 bitbot Logged on 2023-02-03 14:12:05 asciilifeform[jonsykkel]: i.e. joker can try to use his l1 as a 'ddos cannon'
18:41 asciilifeform concretely, instead of a single binary direct msg 'gimme lubystream of hash H' being answered immed. by stream, the station will answer 'sure, but 1st send me back these rng bytes, stream will go to ip they get send back from and it has to be the same 1 from which you had asked just nao'
18:44 asciilifeform no need to recognize, as such, change in AT; simply insist that the addr to which luby piss is to go is able to receive, as well as send, under the requester's key
18:45 asciilifeform you can't forge, lol, ~receiving~ (or rather it aint called 'forging' anymoar if you can..)
18:50 asciilifeform is very similar to the handshake spec already has for rekey, the purpose of which is to prevent a buggy, sad-rng'd, or saboteur peer from forcing the peering key to an apriori-known value
18:53 asciilifeform ( there's naturally no way to prevent an idjit or traitor from divulging key to third party. but he can't force it to e.g. zero )
18:58 asciilifeform ... this may be the 1 and only original idea, lol, in pest protocol. (tho for all asciilifeform knows, even it existed somewhere. but nfi where)
18:58 asciilifeform contrast with e.g. ssl's 'null cipher'(tm) magick bit.
~ 1 hours 14 minutes ~
20:12 phf right, i was over engineering things in my head. simply "guarding" luby with a handshake is good enough
20:14 phf in totally unrelated can someone explain the china weather balloon story to me? i think ~that~ finally broke my clown world processing machinery
~ 35 minutes ~
20:49 crtdaydreams ^ second that, nfi where "spy balloon" nonsense origniated, but from a cursory glance anyone with a modicum of intelligence would observe that a) why worry about the bitch on the porch when there's a bear in the livingroom? and b) why not satellite imagery? lmao
20:52 * crtdaydreams recalls a public satellite imagery website where could view .hk images and naviage around, could zoom in far enough to make out peoples faces quite easily
20:54 crtdaydreams to the extent that faces etc. would be higher quality than found on e.g. passport
20:55 phf that's not the angle i'm even interested in, more like "why is this such a big deal" and "why is there no sane grown up reaction"
20:56 phf 􏿽1950s solution, announce that it's a weather balloon, immediately mount a televised recovery operation, have spooks look over it on a tight schedule, have a pr transfer of the balloon the chinese event, "our slanty eyed best partners seem to have lost their toy, we give it back, to
20:56 phf 􏿽 bright future!" everyone drinks champagne and smiles
20:56 phf 2023 zoomer solution "ugh awkward ugh we're monitoring *shuffles awkwardly around*"
20:56 signpost yeah, the lack of theatric message-sending is bizarre.
20:56 phf w t f
20:57 signpost phf: I saw earlier that chinese state media of some kind was laughing at us being unable to control our airspace, but failed to find it a second time
20:58 asciilifeform balloons aint obsolete, 'cheap & angry' if conspicuous snoop camera
20:58 asciilifeform Official excuse for not popping it, apparently: 'could cause property damage on the ground' lol
20:58 signpost curious if it's scanning radio
20:59 phf the lack of any kind of sane grown up reaction is bizarre. i mean predictable, but that's like the ultimate failure of "hello yes hello" sanity check on there being present any kind of intelligence in the decision making aparatus
20:59 signpost wifi, bluetooth, etc for exploitation
↖
20:59 signpost maybe they want to dump it in the ocean and recover
20:59 asciilifeform hard to imagine wai else to bake such a large one if not to carry luxurious antennae
20:59 crtdaydreams can't knock it down because "Das Racist!!1111!"
21:00 crtdaydreams lolol
21:01 phf signpost, in an obviously public event like that one ought to signal an intent, whatever the strategy is
21:01 signpost sadder answer would be that biden's crackhead son relayed a threat from the chicoms.
21:01 * asciilifeform pictures that there's likely a usg 'golden toilet' wunderwaffe designed to steal balloons, but not actually worx, and officials scheming to blame one anuther
21:01 signpost and then he made up a reason to order no action.
21:02 signpost mil could be internally wtf-ing at grandpa's order.
21:02 signpost not claiming certainty, but any answer suggests woeful unpreparedness for war.
21:06 crtdaydreams too busy waging war on own denizens
21:08 phf so kommuniti consensus is wwiii?
21:10 crtdaydreams phf: doubtful imo
21:10 phf somehow in my world view chinese lost a balloon, "weather balloon" for diplomatic purposes, would've been mildly embarrassed but u.s. saved them face by being even more awkward and incompetent at diplomacy
21:13 signpost chatter says the thing flew over a lot of air force bases.
21:13 signpost and that it was maneuvering.
21:14 signpost https://www.youtube.com/watch?v=g3iFJpGJiug << e.g.
21:14 signpost whiteman's a nuke base, b-2s etc
21:14 signpost I think they scanned our MSFT-running trash.
21:17 phf is it being piloted by one hide ping sam, the infamous ghost of beijing?
21:19 signpost poor guy's instructed to jump out in the atlantic and start drinking.
21:25 phf antenas are for show, hide ping sam remembers every detail of enemy installations by heart
~ 17 minutes ~
21:42 crtdaydreams lol
~ 38 minutes ~
22:21 asciilifeform http://logs.bitdash.io/pest/2023-02-03#1022390 << bluetooth/wifi from 24e3 km is a bit lul imho
22:21 bitbot Logged on 2023-02-03 20:59:16 signpost: wifi, bluetooth, etc for exploitation
22:21 asciilifeform err, 24km
22:34 signpost lol, I somehow pasted the spaceballs thing again.
22:34 signpost link was meant to be a senator twisting his nipples about whiteman base
22:35 signpost ironic name, at that.
← 2023-02-02 | 2023-02-04 → ↑

Random(pest) | Download hourly DB snapshot | Get Source Code

Theme: Light | Dark | System Default