Programming thread

  • Want to keep track of this thread?
    Accounts can bookmark posts, watch threads for updates, and jump back to where you stopped reading.
    Create account
Rango Dango said:
it wouldnt work since you cant really make the row bigger in neovim as far as i know unless you turn on column wrapping but ehhhhhhhhhhhhh that would be very hack-y and wouldnt look good
Click to expand...
ah fair, I was working off the old method for terminal interfaces where you more or less do this.

clear screen
draw
clear screen
repeat.
 
General Zoider said:
ah fair, I was working off the old method for terminal interfaces where you more or less do this.

clear screen
draw
clear screen
repeat.
Click to expand...
i mean there technically is image.nvim with 3 ways to display images
1. kitty's terminal graphics protocol which only works in the kitty terminal (i.e. not on windows)
2. ueberzug which is also linux only and i dont know anything about it but apparently works on every terminal
3. sixel which does work on windows but i have no idea if it would extend rows or it would be strictly for viewing individual images (its also utterly slow apparently)
 
Private Tag Reporter said:
I wanted to write a rebuttal to the fat guy but I couldn't come up with anything good because his arguments are so retarded. I like data-oriented, it's a nice tool to use and to think about problems. It's not a magic bullet. Being honest, the only reason any of that matters is because imperative programming is an ungodly mistake invented by a crazed homosexual that is just a smidgen more convenient to emulate with electronics when in a sane timeline we would have lisp machines and Holy-Scheme rather than Holy-C.
Click to expand...

Bruh. If I knew how to program in Lisp, do you really think I'd be here trying to teach 20+ year web "devs" what a socket is, or senior C++ programmers how to manage memory?

No, I'd be on a yacht full of supermodels because I would have taken over the Universe.

I'm just intelligent enough to know how powerful Lisp is, and just retarded enough to not wield it's power.
 
Speaking of resource management, I had to implement a database pool for a Scheme project recently.

I was adding a healthcheck to a REST API I was working on and I realized that my code couldn't recover if the database went down. I don't think it crashed either, it just permanently kept returning 500s.

And then I read the fine print on the postgres binding I was using and it says "not threadsafe". Chicken's (my Scheme of choice) threading library (srfi-18 compatible) implements green threads, not OS threads. And I wasn't sure if I had to allocate separate connections or not.

Like if I'm not using OS threads, there's zero chance a single call using the database handle will shit the bed. But I don't know if that applies to result row pointers and things like that.

So to be safe, I'm implementing a database pool. I have to fuck around with mutexes and threads and that shit scares the shit out of me. It's so easy to fuck up and they're the kinds of phantom bugs where everything can be fine for months, and then you have three showstopping deadlocks in a row, and then everything's fine again. Horrible to debug.

That's why I really like the higher level abstractions. Like Go's goroutines and channels make concurrency much, much harder to fuck up drastically.

Anyway, I had to do it anyway. It's simple enough and the risk of a deadlock is pretty low with how I planned to use this.

Code: 
(define current-db-pool (make-parameter #f))

(define db-pool-size (make-parameter 32)) ;; let's make this max size

(defstruct db-pool
  url
  (mutex         (make-mutex 'db-pool))
  (cv            (make-condition-variable 'db-pool))
  (available     '())
  (opening-count 0)
  (all           '()))

(define (db-pool-spots-to-allocate pool)
  (- (db-pool-size)
     (+ (length (db-pool-all pool)) (db-pool-opening-count pool))))

(define (open-db-pool url)
  (let ((handles (map (lambda (_) (open-db-handle url))
                      (iota (db-pool-size)))))
    (make-db-pool
     url:       url
     available: handles
     all:       handles)))
Easy enough place to start.

I'm using Chicken's defstruct library. I think Common Lisp had something like defstruct. Like there's a bunch of little libraries for Chicken (and probably most Schemes) that are just lifting features they miss from CL. I have never used CL myself but I bet I know it pretty well just because of cultural osmosis.

So a db-pool has the database URL. That just gets passed to a simpler open-db-handle function I defined elsewhere. It also has a mutex, a condition variable (basically a threading signal variable), a list of available handles, a list of all the allocated handles, and a count for how many connections are currently pending.

The open-db-pool function starts off by allocating all the maximum number of connections it's configured for. I'm not sure if I want to do it this way, or initialize it to 0 and let it grow as needed. Either way works.
Code: 
(define (db-pool-abandon!/locked pool handle)
  (db-pool-available-set! pool
                          (filter (lambda (other) (not (eq? other handle)))
                                  (db-pool-available pool)))
  (db-pool-all-set! pool
                    (filter (lambda (other) (not (eq? other handle)))
                            (db-pool-all pool)))
  (handle-exceptions
      exn
      (log-error "db-pool-abandon!/locked: failed to close connection: ~s"
                 (condition->list exn))
    (wrapped-db-handle-disconnect! handle)))
This is a helper that becomes important later. It's solely an internal function and it's supposed to operate on database pools that are already locked, which is why I named it db-pool-abandon!/locked.

Basically if a connection has shat the bed, this function removes it from the pool. It removes it from both lists and attempts to close the connection, logging any failures that come up.

Next is the real meat of the problem, the db-pool-acquire! function:
Code: 
(define (db-pool-acquire! pool)
  (let ((mx (db-pool-mutex pool))
        (cv (db-pool-cv pool)))
    (mutex-lock! mx)
    (let iter ()
      (let ((avail (db-pool-available pool)))
...
Ok, this is a big function ,but to start off, we grab the mutex and the condition variable and lock the mutex. And then we grab the list of available connections after the mutex is locked. If you're not familiar with Scheme, (let NAME (...)is a combination of let, which creates local variables, and a recursive function. It's a pretty common way to loop.

So when we loop, we'll just call it with (iter).

Next we have a few possible cases for our db pool state:
Code: 
        (cond
         ;; if there's an available connection and it's working
         ((and (pair? avail) (wrapped-db-handle-healthy? (car avail)))
          (db-pool-available-set! pool (cdr avail))
          (mutex-unlock! mx)
          (car avail))
So cdr is an old lisp thing, and it means "take the rest of the list minus the head". Car means head, cdr means tail. So in this case, if the available list isn't empty and the first connection is working, grab it, unlock the mutex and return.
Code: 
         ;; there's an available connection and it's not working
         ((pair? avail)
          (log-error "db-pool handle isn't connected, abandoning it")
          (db-pool-abandon!/locked pool (car avail))
          (log-debug
           "db-pool-acquire! lost connection, dropping; (length all)=~s"
           (length (db-pool-all pool)))
          (mutex-unlock! mx)
          (mutex-lock! mx)
          (iter))
Next possibility: the available list isn't empty, but since it didn't pass the healthy check earlier, we need to remove this connection.

I'm logging just for testing purposes.

Also, note we unlock the mutex and immediately relock it. This is an opportunity for another thread to hop in here.

And as you can see, we call iter and recurse back to the top of the function again.
Code: 
         ;; we're at our maximum connection count
         ((<= (db-pool-spots-to-allocate pool) 0)
          (when (negative? (db-pool-spots-to-allocate pool))
            (log-warning "db-pool-allocate!: overallocated; len=~s; opening-count=~s; db-pool-size=~s"
                         (length (db-pool-all pool))
                         (db-pool-opening-count pool)
                         (db-pool-size)))
          (mutex-unlock! mx cv)
          (mutex-lock! mx)
          (iter))
Makes enough sense. Although note: see mutex-unlock!? I'm passing the condition variable alongside the mutex. mutex-unlock! optionally takes that extra variable and if you provide it, it'll wake up any threads sleeping on that condition variable before unlocking the mutex.

As I write this, I wonder if I should've done that the earlier time I called it too. I cannot remember my rationale now. Threading is hard.

And finally, the last case is when none of the other cases apply:
Code: 
         ;; we're not at our max, allocate a new one
         (else
          (log-debug "db-pool-acquire! allocating new connection")
          (db-pool-opening-count-set! pool (add1 (db-pool-opening-count pool)))
          (mutex-unlock! mx)
          (let ((handle
                 (handle-exceptions
                     exn
                     (begin
                       (log-error "db-pool-acquire!: couldn't allocate new connection: ~s"
                                  (condition->list exn))
                       (mutex-lock! mx)
                       (db-pool-opening-count-set! pool (sub1 (db-pool-opening-count pool)))
                       (condition-variable-signal! cv)
                       (mutex-unlock! mx)
                       (abort exn))
                   (open-db-handle (db-pool-url pool)))))
            (mutex-lock! mx)
            (db-pool-all-set! pool (cons handle (db-pool-all pool)))
            (db-pool-opening-count-set! pool (sub1 (db-pool-opening-count pool)))
            (condition-variable-signal! cv)
            (mutex-unlock! mx)
            handle)))))))
So a lot to unpack here but it's not too complicated. We increment the number of connections in progress. I specifically didn't want a sluggish connection request to hold all the threads up.

This is a subtle thing that's easy to fuck up. So I increment the count (so other threads don't overshoot our max connection count, see how db-pool-spots-to-allocate is defined above), then unlock the mutex and while it's unlocked, try to connect to the db.

So this is kind of a pain in the ass to read, but we try to open the connection with open-db-handle. above that is the exception handler for if that fails. What I do in the failure case is catch the exception, decrement the opening-count for the pool, trigger the condition variable, unlock the mutex, and then re-raise the exception with (abort exn).

The happy path is below that. With the happy path, we re-lock the mutex, and then we add our newly opened connection to the list of total connections, signal the condition variable, unlock the mutex, and then return the handle.

Whew!

When I first wrote this, I mistakenly added the new handle to the available list as well. I only caught it by accident when I was debugging something else.

And then finally, the much simpler release function:
Code: 
(define (db-pool-release! pool handle)
  (let ((mx (db-pool-mutex pool))
        (cv (db-pool-cv pool)))
    (mutex-lock! mx)
    (if (wrapped-db-handle-healthy? handle)
        (db-pool-available-set! pool (cons handle (db-pool-available pool)))
        (db-pool-abandon!/locked pool handle))
    (condition-variable-signal! cv)
    (mutex-unlock! mx)))

There's some other helper functions, but that's the gist. So anywhere in the code can go (with-db-handle (lambda () (printf "value from db is ~s\n" (db-exec-first-result (db-handle) "SELECT 1")))).

Safir said:
The real answer is Docker caches each intermediate step of the Dockerfile and can then reuse it if it's the same between builds; if your build changes a little, you don't have to rebuild everything from the raw image. So if you write some code and do "copy fucking everything", it's a cache miss because the previous version of "everything" is different from the current version (you just wrote some new code) and it's going to download 100s of Python libraries and potentially 1000s of javascript libraries. But if you do "copy requirements.txt ." (or whichever file has the list of libraries), install the libraries, then copy over the rest of the code, the intermediate cached image with the installed libraries will be then reused.
Click to expand...
Yeah, my dockerfiles are always a lot longer and more thought out than my coworkers' for exactly this reason.

It's a useful tool for plenty of purposes beyond just process isolation. But you need to actually understand it and use it for its strengths.

I can spend all day fiddling with all kinds of stupid database settings and environment variables and all kinds of bullshit. By the end of the day, I get the code working and I'm pretty sure I documented everything I did and the code should work in prod.

And before I fuck off for the day (to continue to work on code I actually like), it's super convenient to be able to run isolated tests in a clean environment. And the difference between that taking 20 minutes versus 2 minutes is if the Dockerfile wasn't written by a moron.
 
Fat Camp Intern said:
I'm just intelligent enough to know how powerful Lisp is, and just retarded enough to not wield it's power.
Click to expand...
Marvin said:
Speaking of resource management, I had to implement a database pool for a Scheme project recently.
Click to expand...
1778233394813.png
 
man having a non-us layout sucks for emacs
especially the thinkpad keyboard with the small fn in the bottom left and big ctrl next to it that i have to swap the function of in the bios because its mad annoying ottherwise

and the non us layout sucks because i dont have a right alt
M-x? no nigga you get ź
M-:? no nigga you get a dead key (equivalent to a tilde on windows)
M-v? „

neovim lowkey better because of that
 
Rango Dango said:
man having a non-us layout sucks for emacs
especially the thinkpad keyboard with the small fn in the bottom left and big ctrl next to it that i have to swap the function of in the bios because its mad annoying ottherwise

and the non us layout sucks because i dont have a right alt
M-x? no nigga you get ź
M-:? no nigga you get a dead key (equivalent to a tilde on windows)
M-v? „

neovim lowkey better because of that
Click to expand...
Non-Us just plain sucks for programming.
 
Rango Dango said:
and the non us layout sucks because i dont have a right alt
Click to expand...
Setxkbmap to something normal? Slightly OT, but recent update deleted my custom keyboard layout, because I didn't know you shouldn't just place it into /usr/share/. Oh well, you live and you learn.
 
canalCrab said:
Non-Us just plain sucks for programming.
Click to expand...
theres no real difference outside of that tbh
i have a 105-key keyboard on my thinkpad and the key inbetween shift and z (horrifying i know) is <> on non-uk layouts (which is mad useful for emacs actually) on a uk its the \| key (the one above enter on a normal keyboard)
Zeftax said:
Setxkbmap to something normal
Click to expand...
i guess
i use xfce so i can have two layouts and switch between them with just a keybind but look at all these symbols i have access to πœ©ß←↓→þł…ə’ŋæð„”µ≤≥~¨~≠²³¢€½§·«
even not counting the fact that its a 105-key this keyboard just plain sucks
the fn-ctrl key thing ive already talked about
second win key is replaced by a FUCKING PRINT SCREEN
pg up and down sandwich the up arrow key which at first does sound pretty nice i do accidentally press one of them when i want to press the up arrow and it is annoying
the escape key is very wide for some reason
and uh yeah im the most titled by the prtscr because i press it accidentally when i want to press alt gr
 
Last edited:
Rango Dango said:
switch between them
Click to expand...
Yeah I lived like this for a long time, but keeping the "state" of the keyboard in my mind was kind of annoying, especially if I wasn't just sitting at the computer, I sometimes even typed something out with the wrong keyboard, which sucks. As I hinted at, I recently made a custom layout that incorporates the symbols I use on top of the us keyboard (using alt gr, sometimes shift). I'd send it for inspiration, but portage overwrote it :D.

Spoiler: some links I have saved from when I was looking up how to do it.
https://wiki.gentoo.org/wiki/Keyboa...g_system-wide_configured_layouts_and_shortcut
https://people.uleth.ca/~daniel.odonnell/Blog/custom-keyboard-in-linuxx11
https://www.charvolant.org/doug/xkb/html/index.html
https://www.tsu.ru/~pascal/en/xkb/
https://xkbcommon.org/doc/current/group__compose.html

Spoiler: My own notes, short.
The main config file is in /usr/share/X11/xkb/symbols/<base lang>, I started with the cz lang and looked at the programmers variant for inspiration.
This file can get overwritten by updates, the right place is apparently in /usr/local/share.
Names of symbols are in /usr/include/X11/keysymdef.h.
Names of key names are in /usr/share/X11/xkb/keycodes/evdev -- use this with xev.
 
Rango Dango said:
so i can have two layouts and switch between them
Click to expand...
nvm i cant
when i press alt gr x on the us layout while having the polish layout in the background it just inserts x instead of triggering M-x
i have no idea why it would work that way
Zeftax said:
Yeah I lived like this for a long time, but keeping the "state" of the keyboard in my mind was kind of annoying, especially if I wasn't just sitting at the computer, I sometimes even typed something out with the wrong keyboard, which sucks. As I hinted at, I recently made a custom layout that incorporates the symbols I use on top of the us keyboard (using alt gr, sometimes shift).
Click to expand...
personally i really like having default things, im not a customization fan
and this way i dont need to fight my muscle memory when i have to use the default layout
so so far (neo)vim is better for this keyboard because the modifier keys are way too annoying to access
 
Fat Camp Intern said:
Using lisp for a database pool is retarded. It's even more retarded with mutexes.
Click to expand...
Doing synchronization outside of the language would be doable, but that'd freeze up the whole process. Green threads vs OS threads.

I wanted to keep the whole project in one language when possible. That would mean playing nice with srfi-18 threading, which means mutexes.

I'd have preferred if there was a library for this. But it's not so terrible doing it myself. I'm just nervous about the concurrency. But beyond the concurrency, it's just pretty simple pointer bookkeeping.
 
You must log in or register to reply here.
Back
Top Bottom