Coal All The Way

[coalcube]

It's coal-bo time.

 Coal Friending Meme


Yuleporn | Make the Yuletide Gay / TNB+ | Femslash Festivus 
YuleBuilding | Two for One | Crueltide | Misses Clause
YuleSwaps | IF | Wrapping Paper 

Yuletide Discord for Hippos. Google Group for PHs. F_F wiki for history.

yuletide | yuletide_admin | Writing Post
2019 Collection | 2019 Madness
2019 App | Letter Post (LJ) 
 

Flat / Recent | Top-Level (Last) | Search

Comments

Re: Treatless App Draft

[(Anonymous)]

CYRT

Is it? It's working for me, though it does take forever. Do you get a 500 error or just stalling out?

Re: Treatless App Draft

[(Anonymous)]

Application error, 503 POST error to https://treatless.herokuapp.com/search

Feels like a timeout to me, since spinny spinny, but I imagine you're doing quite a bit of paging work in the background so no real idea.

Re: Treatless App Draft

[(Anonymous)]

SC

Ah, your example has Critmas capitalised; it works with lowercase and only lowercase, methinks.

Re: Treatless App Draft

[(Anonymous)]

On this note, welcome to my life. I'm so happy to see another coder around, flailing at AO3's lack of API in the same cesspool of suffering as me :3

Re: Treatless App Draft

[(Anonymous)]

CYRT

Ah, my logs actually say Too Many Requests error, will somehow have to handle that gracefully at some point. I TRULY DON'T KNOW IF THERE'S A BETTER WAY TO DO THIS THAN SCRAPING EVERYONE'S GIFT PAGES, AO3 please throw me a fucking bone (and by bone I mean "can it be harder to query for a collection any given fic is in").

I don't know how you're handling the massive amount of data with e.g. Yuletide requests, did you just download it all one time so it's all in a DB on the back end?

Re: Treatless App Draft

[(Anonymous)]

This, my dear friend, is why I haven't set up a dynamic spin to win click to generate your own exchange app meta-app yet...

With closed signups that only reveal prompts after like Yuletide, yes. I dump the tagset before signups, then dump the prompts when they are revealed.

With the running signup exchanges like ToT and CB, I leave my computer turned on because I am cheap as hell and run the scraping as a cron job. It is a sad trumpet life.

Every time I get close to getting a meta-app stood up, I lose steam because I'm mostly tired from coding 24/7 in real life. Maybe this year... I'm pretty close. Just need to stand up an actual backend server for the running signups exchanges.

Sans API, to the main point again, there's really no good way other than working smart. I page incrementally so I don't lose data. You might want to chuck already-scraped exchanges into a backend and only update them max once idk every 15 minutes or so.

Re: Treatless App Draft

[(Anonymous)]

And yeah, with unrevealed exchanges AFAIK you are shit outta luck. I'm better off with Yuletide and some of the other exchanges I help with because mods are willing to give me mod rights so I can at least get the works pages to show usernames, but you have to build a certain level of trust with them that you won't go on a Delete All Assignments accidental click rampage.*


* No shade to Morbane, I literally have had nightmares of accidentally coding the app and blowing up Yuletide because I fatfingered something in the code. I am very, very, VERY paranoid.

Re: Treatless App Draft

[(Anonymous)]

NC

You can get a list of someone's gifts in a particular collection with a URL like this one:

https://archiveofourown.org/users/{LOGIN}/gifts?collection_id={COLLECTION}

(The page itself doesn't look right, so I suspect it's not supposed to exist. But it does, so I personally use it to count gifts. Not sure if it'd be an improvement over the normal gift pages for you, but I thought I'd mention it since you mentioned querying for which collections a fic is in.)

Re: Treatless App Draft

[(Anonymous)]

CYRT

Ooooh, I can use this to count gifts for revealed collections faster, that's really useful! Thanks coalie!