I just created the PR for the caching ticket (#27). But, unfortunately, I
found that there was an issue with the pages table; i.e., the hashes were
being stored without padding out the hash value to 32 characters. The
caching code needs the hash to be exactly 32 characters long so it can
build out the caching directory. The directory structure takes the hash and
creates three directories: one for the first 8 characters, one for the next
8, and the third for the next 8. Then the cache filename is the last 8
characters. So that's going to break if your pages table contains hashes
with less than 32 characters.
I can decline the PR I pushed and add a migration to pad out any existing
records so you won't have to delete anything, then resubmit the PR. I'd
rather do that than have you need to delete your data.
On Sat, Dec 7, 2019 at 4:23 PM bareheiny <dmarc-noreply@xxxxxxxxxxxxx>
wrote:
My library is now loaded (hopefully for the last time)....no metadata
embedded, but CX has picked up the series etc. from the file name – where
the name makes sense.
I’ll start having a look at load times again soon – as well as doing some
scraping, as I expect the more metatdata that’s available the slower the
load will take.
I’m likely repeating myself...but I really do think caching the covers
will significantly decrease the library page load times.
Which just leaves the metdata – being loaded into the donut chart. Need
to figure out a way to measure that properly though....”I think it’s slow”
doens’t really cut it >_<
*From: *Darryl L. Pierce <mcpierce@xxxxxxxxx>
*Sent: *Sunday, 8 December 2019 4:02 AM
*To: *comixed-dev@xxxxxxxxxxxxx
*Subject: *[comixed-dev] Re: The issue of fetching the library
@bareheiny - I think ultimate you're the guy I'll need to lean on to find
the optimal solution for this problem.
On Sat, Dec 7, 2019 at 9:54 AM Darryl L. Pierce <mcpierce@xxxxxxxxx>
wrote:
So I'm working on re-enabling the multi-comic scraping today (the code's
nearly complete) and I'm thinking about comic selection. Specifically, how
the "Select All" button is no longer really doing that; i.e., since we
don't have the whole library in memory, there's no way to select "all". You
can only select, at best, 100 comics depending on the number of comics you
display per page.
This has me thinking that neither the current model (downloading only a
page at a time) or the previous model (downloading the full library) is the
right answer. So I'm looking for ideas or suggestions for how we can do
this better.
My first idea is re-enabling the constant background update, but limit it
to only returning high-level details for the comics (id, publisher, series,
issue #, characters, teams, locations, stories) (effectively, that's
everything that makes up the collections). I thing that would go much
faster that the previous downloading of the full set of details for each
comic.
On the backend, though, it still takes same amount of work to fetch the
data, but should be much faster to marshal it for the response since it's a
small number of fields (and no page details).
Any thoughts, ideas or suggestions?
--
Darryl L. Pierce <mcpierce@xxxxxxxxx>
"Le centre du monde est partout." - Blaise Pascal
"Let's try and find some point of transcendence and leap together." - Gord
Downie
--
Darryl L. Pierce <mcpierce@xxxxxxxxx>
"Le centre du monde est partout." - Blaise Pascal
"Let's try and find some point of transcendence and leap together." - Gord
Downie
