#ossasepia | 2020-02-29

00:55

thanks whaack, I had some fun with that.

01:04

what occurred to me for your present fleet memory issue: I would imagine sbcl has a way to run the garbage collector explicitly and capture the stats programatically (I expect there's only one of interest really, "how much heap is in use?" - not sure where that is exactly in the detailed output.) Make a loop interleaving that with whatever operation you want to test - joining a network perhaps -

01:04

jfw

or run it periodially in a thread, logging to a file. See how it evolves over time. This is a rough sort of approach; maybe sbcl has some fancier memory profiling. Or more study of your own + cl-irc code couldn't hurt.

01:11

jfw

really it seems to me there should be epsilon change in memory usage with respect to time for a given number of networks, since messages are logged externally.

01:15

jfw

also you could run a local ircd ("ratbox" came up as one of the supposedly solid options last I looked into this) to facilitate a large number of connections / other noisy tests.

~ 2 hours 43 minutes ~

03:58

diana_coman

http://logs.ossasepia.com/log/ossasepia/2020-02-28#1019734 - what's wrong with enjoying learning and/or puzzles? re hardware though, it's a sort of once-removed to learn about it via tools on top and it's unclear why would you go that route at all.

03:58

ossabot

Logged on 2020-02-28 16:52:44 whaack: diana_coman: tbh I have a concern that the reason I want to learn the cmd line tools is in part because I enjoy solving little puzzles. Otherwise I would say I want to learn them because I see them as a means to increase my productivity and I guess also (this could be added to my list of whys) as a way to learn about topics such as networking and how various parts of a computer work (tools can

03:59

diana_coman

http://logs.ossasepia.com/log/ossasepia/2020-02-28#1019730 - nice, well done jfw ! and indeed, not having a republican block explorer sucks.

03:59

ossabot

Logged on 2020-02-28 16:45:27 jfw: In other news, a gbw transaction escapes into the wild: https://www.blockchain.com/btc/tx/7386679d841bdca0b7c412d09d936627e7310597542be8f7d75cb39811d799b1

~ 5 hours 19 minutes ~

09:19

00:55	jfw	thanks whaack, I had some fun with that.
01:04	jfw	what occurred to me for your present fleet memory issue: I would imagine sbcl has a way to run the garbage collector explicitly and capture the stats programatically (I expect there's only one of interest really, "how much heap is in use?" - not sure where that is exactly in the detailed output.) Make a loop interleaving that with whatever operation you want to test - joining a network perhaps -
01:04	jfw	or run it periodially in a thread, logging to a file. See how it evolves over time. This is a rough sort of approach; maybe sbcl has some fancier memory profiling. Or more study of your own + cl-irc code couldn't hurt.
01:11	jfw	really it seems to me there should be epsilon change in memory usage with respect to time for a given number of networks, since messages are logged externally.
01:15	jfw	also you could run a local ircd ("ratbox" came up as one of the supposedly solid options last I looked into this) to facilitate a large number of connections / other noisy tests.
		~ 2 hours 43 minutes ~
03:58	diana_coman	http://logs.ossasepia.com/log/ossasepia/2020-02-28#1019734 - what's wrong with enjoying learning and/or puzzles? re hardware though, it's a sort of once-removed to learn about it via tools on top and it's unclear why would you go that route at all.
03:58	ossabot	Logged on 2020-02-28 16:52:44 whaack: diana_coman: tbh I have a concern that the reason I want to learn the cmd line tools is in part because I enjoy solving little puzzles. Otherwise I would say I want to learn them because I see them as a means to increase my productivity and I guess also (this could be added to my list of whys) as a way to learn about topics such as networking and how various parts of a computer work (tools can
03:59	diana_coman	http://logs.ossasepia.com/log/ossasepia/2020-02-28#1019730 - nice, well done jfw ! and indeed, not having a republican block explorer sucks.
03:59	ossabot	Logged on 2020-02-28 16:45:27 jfw: In other news, a gbw transaction escapes into the wild: https://www.blockchain.com/btc/tx/7386679d841bdca0b7c412d09d936627e7310597542be8f7d75cb39811d799b1
		~ 5 hours 19 minutes ~
09:19	whaack	jfw: http://logs.ossasepia.com/log/ossasepia/2020-02-29#1019741 Right. I don't think that the memory used is growing directly with the number of messages logged, since the bot that is still running has logged over 3x as many messages as the one that crashed.
09:19	ossabot	Logged on 2020-02-29 02:27:57 jfw: really it seems to me there should be epsilon change in memory usage with respect to time for a given number of networks, since messages are logged externally.
		~ 21 minutes ~
09:40	whaack	When I use the (room) function on my bot that is still running I can see that the memory usage fluctuates. There may be some non negligible memory being used when a large number of incoming messages occur all at once. If I am receiving messages faster than I can write them to the postgresdb for a long enough period of time, that would cause my program to run out of memory.
		~ 15 minutes ~
09:56	whaack	in interesting weird found in my irclogs http://bitcoinshell.mooo.com/
09:59	whaack	the owner of the site offers a VM with 128MB of disk space to anyone, but to create an account on the vm you need to beat his bot in a telnet game of pong.
10:02	*	whaack tried but gave up since it takes a bit of time to secure the victory.
		~ 52 minutes ~
10:55	whaack	diana_coman: there's nothing wrong with enjoying learning or solving puzzles. but if that's the only/primary reason to learn something then maybe one's learning what they want to learn rather than what they need to learn.
		~ 1 hours 10 minutes ~
12:05	jfw	thanks diana_coman!
12:06	jfw	whaack: looks like what I had in mind can be done by the entirely undocumented (sb-kernel:dynamic-usage), following a (gc :full t).
12:08	jfw	Certainly the database will need to be able to keep up, there may be some postgres tuning to do if that's really the problem. But does your code really queue up messages from the network before processing them? Why?
		~ 23 minutes ~
12:31	whaack	jfw: Any message queueing that may be happening is unintentional. I believe that the code waits for the db write to finish before calling "select2" to read the next line. My guess is that incoming messages that have not been "read"'d take up some space in the sbcl process, and if as you say the database can't keep up, those incoming messages waiting to be read and then written to the db pile up
12:31	whaack	and cause an OOM error
		~ 26 minutes ~
12:58	jfw	whaack: assuming these select2/read are ~equivalent to the corresponding system calls, then no; there's some limited buffering done by the kernel; when that fills, tcp flow control will stop the server from transmitting until more space is made.
13:08	whaack	jfw: ah thank you, that is very good to know.
13:20	jfw	yw.
		~ 53 minutes ~
14:13	diana_coman	BingoBoingo: lol, that is any way better but the question was re previous non-approach
14:13	ossabot	(agriculturalsupremacy) 2020-02-29 BingoBoingo: thimbronion: Which story would you rather write, the chinese spy scoop you've been weighing all week or the US surrendering to the Taliban?
14:13	ossabot	(agriculturalsupremacy) 2020-02-28 BingoBoingo: jfw: Ty, I'll leave this open for 5 hours to see if any takers claim it while I do Spanish class and work on refining the plan.
14:13	diana_coman	did that older possible story fade entirely given the great handling ?
14:15	BingoBoingo	It didn't, it's still a hole in coverage, less fresh and still open.
14:17	diana_coman	BingoBoingo: yes but specifically what are you going to do? and moreover, you've got now to see that there is this sort of potential situation - and it's a good one ie kuddos to jfw for the lead there! - so don't ignore it/sleep on it because it's not yet biting you hard enough or something; plan ahead so it does NOT end up biting you harder.
14:22	BingoBoingo	diana_coman: Alright, I'll cover it today. I'd been trying not to write myself for Qntra before 19:00 hrs UTC to keep it from consuming the bulk of the day, but it seems pretty clear that when these leads come in I'm going to have to carve out some time to hammer a piece, then move whatever got displace into whatever time I'd reserved to searching and hammering out an unknown story later.