[GeoStL] slowdown

            Here's some info on the problems with the slowdowns. Taken form the 
forums tonight. They didn't indicate that the "mining" would have to stop, just 
that that is what causing some of the slowdown and they are working on a 
solution.

            Rich


            Database overload 

            The message above is a database timeout error. It occurs when the 
web site makes a request to the database and the database doesn't respond in 
the expected amount of time.

            This is partly due to the increased traffic on the web site, but is 
mostly due to people using automated tools to suck down data from the site in 
bulk. The load placed on the database when these tools run is astronomical, and 
basically causes a denial of service to other geocachers. This is the reason we 
work to prevent automated tools from accessing the site.

            We're working to optimize the database and some of our code to make 
it more efficient. Also, our efforts to convert the site to the .NET platform 
will help as well. We're working hard to address these performance issues with 
the site and are confident that we'll come up with a scalable long-term 
solution.

            -Elias 
--------------------------------------------------------------------
            Posts: 156 | From: Seattle, WA USA | Registered: September 02, 2000 
     
            UtahJean 

            Geocaching Supporter  posted August 26, 2002 06:44 PM August 26, 
2002 06:44 PM    
--------------------------------------------------------------------
            Slowdown. 

            Would it help if we backed off using the new Pocket Query 
Generator? 

            I can't get on the most recent logs page or the list of new caches 
for my state, both of which are necessary for outwitting other players of the 
sweet little cach-u-nut game we have here in Utah. 
--------------------------------------------------------------------
            Posts: 7 | From: Utah, USA | Registered: June 07, 2001 
     
            robertlipe 
            Charter Member  posted August 26, 2002 06:51 PM August 26, 2002 
06:51 PM    
--------------------------------------------------------------------

              quote: 
------------------------------------------------------------------
              Originally posted by Elias:
              [ slowdown ] is mostly due to people using automated tools to 
suck down data from the site in bulk. The load placed on the database when 
these tools run is astronomical, and basically causes a denial of service to 
other geocachers. This is the reason we work to prevent automated tools from 
accessing the site. 
------------------------------------------------------------------



            While I don't know the details of the current data flow, of course, 
I find the generalization faulty becuase well-designed "data suckers" can serve 
a multitude of users and actually result in fewer page requests on the central 
servers. As long as the fetches aren't greedy (i.e. they pull only the pages 
that humans would have pulled anyway) and implement caching/sharing so the 
pages are served to multiple users, the total load would be less. (I have no 
way of knowing if the ones that are hammering you are characterized as such or 
not; I'm only offering that some suckers can be your friends.)

            This is why I've asked several times for published guidelines on 
such suckers so that they can hit the servers during times of low load, access 
data in lightweight formats, etc. So far, I've heard no answers.

            Personally, I'm still pinning hopes on GPX. Although this will 
ultimately result in more roams of the database and likely more total bytes 
served (each user will get their own query instead of sharing one) it can 
likely be done by lighter-weight code becuase you don't have do do the 
formatting of the data on the way to HTML. 
--------------------------------------------------------------------
            Posts: 73 | From: Franklin, TN | Registered: December 23, 2001 
     
            Elias 

            Groundspeak Lackey  posted August 26, 2002 07:25 PM August 26, 2002 
07:25 PM    
--------------------------------------------------------------------

              quote: 
------------------------------------------------------------------
              Originally posted by UtahJean:
              Would it help if we backed off using the new Pocket Query 
Generator? 
------------------------------------------------------------------


            No, feel free to use the Pocket Queries as much as you like. We try 
to run the Pocket Queries when the load isn't as high, and we built into the 
code some pretty good caching such that each cache is only queried from the 
database once, no matter how many queries request that cache. 
--------------------------------------------------------------------
            Posts: 156 | From: Seattle, WA USA | Registered: September 02, 2000 
     
            Elias 

            Groundspeak Lackey  posted August 26, 2002 07:45 PM August 26, 2002 
07:45 PM    
--------------------------------------------------------------------

              quote: 
------------------------------------------------------------------
              Originally posted by robertlipe:
              While I don't know the details of the current data flow, of 
course, I find the generalization faulty becuase well-designed "data suckers" 
can serve a multitude of users and actually result in fewer page requests on 
the central servers. 
------------------------------------------------------------------


            I wasn't talking about proxy servers or other "well-designed 'data 
suckers'". I've watched over a dozen AOL proxy servers query the site 
simultaneously. In general, they're pretty good about how they grab pages, but 
its still a reasonable load. However, taking the hit from those is clearly 
orders of magnitude better than having all the users behind them hitting the 
site directly.

            I was really talking about home-brew applications that people build 
to grab huge sections of the data in bulk. Or applications that increment cache 
ids and grab thousands of cache detail pages (using logs=y, of course) as fast 
as we can serve them. It doesn't take very many of these to brutally impact the 
server. And its almost scary as to how many of these are happening right now as 
I type this.

            The issues are really a combination of things: traffic on the site 
continues to grow, more and more automated tools are mining the site, and we've 
got some old code that isn't as efficient as it could be.

            Its just a growing pains issue and we'll sort it out. 

            -Elias 
--------------------------------------------------------------------
            Posts: 156 | From: Seattle, WA USA | Registered: September 02, 2000

Attachment: 10573_200.jpg
Description: JPEG image

Attachment: icon9.gif
Description: GIF image

Attachment: profile.gif
Description: GIF image

Attachment: quote_reply.gif
Description: GIF image

Attachment: edit.gif
Description: GIF image

Attachment: icon1.gif
Description: GIF image

Attachment: 5_100.gif
Description: GIF image

Attachment: icon2.gif
Description: GIF image

Attachment: icon_smile.gif
Description: GIF image

Other related posts: