The frontend is a single-page app; i.e., it doesn't reload the page when
you navigate around, but only loads the data, refreshes the page and
updates the URL. If you reload the page then all of its state is reset and
it has to reload the entire data set from the backend.
In between library update increments, the frontend merges the metadata into
its state to track those relationships. I would guess that's the best area
to start looking to optimize the library loading rather than the overall
loading module. If we can find ways to tighten that up we can make things
more performant.
On Sat, Jan 11, 2020 at 3:53 AM Guy Incognito <dmarc-noreply@xxxxxxxxxxxxx>
wrote:
The library load reset could be my doing – from memory, it occurred when I
manually edited the address in the browser.
The slow loading though...I’ve logged it before. Usually the delay was
between a get 100 starting, completing and then a new get 100 firing.
I’ve suspected that there was something going on after the 100 records
were returned, and that was largely confirmed when I removed all metadata
from my comics – the library loaded quickly, but I struck cover issues
(mostly resolved by your code changes...but adding the metadata back in
helped as well).
So...here are some interesting stats:
Number of comics
47,888
Characters
448,361
Teams
62,495
Locations
78,468
Publishers
214
Series
5,627
A large comic collection for sure, but it’s very disparate – 5.6k series
across 214 publishers. But the character, team and location counts are
pretty high.
So to test a theory, I deleted all records from the characters, teams and
location tables – the library load completed in a reasonable amount of time
(not great, but tolerable and most definitely useable).
Food for thought I hope 😊
*From: *Darryl L. Pierce <mcpierce@xxxxxxxxx>
*Sent: *Saturday, 11 January 2020 2:30 AM
*To: *comixed-dev@xxxxxxxxxxxxx
*Subject: *[comixed-dev] Re: New library update code
Hrm, navigating between pages shouldn't affect the library load or cause
it to reset since it's a background process and no part of the application
can affect it directly or even reach it.
I'm adding logging to the frontend code, and have already fully loaded the
library management code with logging. If you could, would you please open
your browser's developer tools, reproduce this reset and then copy and
paste the logging output from the developer tools console to an email? Then
we'll have some level of traceability into the issue you're seeing.
On Thu, Jan 9, 2020 at 7:53 PM Guy Incognito <dmarc-noreply@xxxxxxxxxxxxx>
wrote:
A related issue – the library is slow to load.
I’m pretty sure I’ve mentioned this before (usually pointing my finger at
the donut chart) – but I’ve been so focussed on the covers loading, that I
didn’t realise the other impacts of the loading process.
At the moment, I’m sitting at a comic count of 21,385 out of around 48,000
– historically I’m timed out of CX before the load is fully complete.
I’ve only just noticed / realised that this also impacts the Collections
pages. For example, 2,183 series are loaded. For now. After a few
minutes, I expect this will get to 3,000 or so.
Obviously I can’t navigate to a specific series (or publisher, team,
location etc. ) if it hasn’t been loaded – at the moment, I can’t find and
navigate to Venom, as the load is only up-to the early Ts.
In some cases, navigating to a different CX page causes the library to
start reloading, putting me back at square one.
Just something to put on the pile of things to look at – eventually :P
*From: *Darryl L. Pierce <mcpierce@xxxxxxxxx>
*Sent: *Wednesday, 8 January 2020 3:42 AM
*To: *comixed-dev@xxxxxxxxxxxxx
*Subject: *[comixed-dev] Re: New library update code
So glad to hear. Once this latest PR I put up this morning is merged I'm
hoping we can put this to be for a while and hopefully get some more users
and developers helping to improve it.
On Mon, Jan 6, 2020 at 9:44 PM Guy Incognito <dmarc-noreply@xxxxxxxxxxxxx>
wrote:
The update is looking good – navigation is a lot better than it was. It
feels like up-to a minute to return a page of 100 covers...but it’s
probably not that high.
Unless there are others with large libraries that can do some testing, I
think for now this can be put to one side – it’s fast enough, but could be
better. But without input from other users there’s no way to determine if
it’s me, my machines or CX.
*From: *dmarc-noreply@xxxxxxxxxxxxx
*Sent: *Monday, 30 December 2019 10:23 AM
*To: *comixed-dev@xxxxxxxxxxxxx
*Subject: *[comixed-dev] Re: New library update code
Merry Christmas and all that 😊
So far, the latest build is looking good as far as navigation goes. In
saying that, I’m only about 12 hours into a four day import process.
I’ll keep an eye on things and report back later.
*From:* comixed-dev-bounce@xxxxxxxxxxxxx <comixed-dev-bounce@xxxxxxxxxxxxx>
*On Behalf Of *Darryl L. Pierce
*Sent:* Sunday, 29 December 2019 8:32 AM
*To:* comixed-dev@xxxxxxxxxxxxx
*Subject:* [comixed-dev] New library update code
Hey, all! Hope you had a good holiday.
I've pushed up the PR for the updated library updating code. It's not as
streamlined as I could make it, only loading the comic (not pages) and
sending the minimum information needed to display comics in lists
(including collections). I honestly can't think of how we could make it
faster at this point.
I removed the paginated loading code since it was, ultimately, going to
make the app far to unwieldy to extend since it required constantly going
back to the server to load data.
Also, testing it locally on my slowest laptop, it's pretting performant
with processing comics being imported and especially with loading covers
and images. I had it load 100 comics/page and it would load the covers a
very short period of time: not instantaneously, of course, but pretty fast.
And I didn't experience any of the covers appearing and then being hidden.
They stayed on the page once loaded.
I'm hoping that this PR can let us put this issue to bed for now and move
on to other features I had hoped to get done for the next release.
--
Darryl L. Pierce <mcpierce@xxxxxxxxx>
"Le centre du monde est partout." - Blaise Pascal
"Let's try and find some point of transcendence and leap together." - Gord
Downie
--
Darryl L. Pierce <mcpierce@xxxxxxxxx>
"Le centre du monde est partout." - Blaise Pascal
"Let's try and find some point of transcendence and leap together." - Gord
Downie
--
Darryl L. Pierce <mcpierce@xxxxxxxxx>
"Le centre du monde est partout." - Blaise Pascal
"Let's try and find some point of transcendence and leap together." - Gord
Downie
