Comments on: Advanced Squid Caching in Scribd: Hardware + Software Used

By: Alek

Alek — Mon, 02 Nov 2009 00:47:34 +0000

Great post, Would you please give us an idea about the perfect ideal configuration for the squid.conf file, memory, cache size etc..

PS: I’m running over a 150 clients.

Thanks

By: Scoundrel

Scoundrel — Sat, 12 Sep 2009 15:12:21 +0000

I don’t mind, what do I need to do to share it there?

By: Chris

Chris — Sat, 12 Sep 2009 08:30:07 +0000

Nice info. I have marked in StumbleUpon.
Do you mind to share the info on http://highscalability.com/

By: Jim

Jim — Fri, 21 Aug 2009 18:33:00 +0000

Hmmm. Seems to be an echo in here. ;]

By: Maneesh Bhati

Maneesh Bhati — Fri, 21 Aug 2009 11:50:22 +0000

Nice. Fast IO makes sense, though I’m puzzled that there was no reasonable RAID config faster than JBOD. Would you mind sharing a bit more around that?

By: Tiranga

Tiranga — Mon, 17 Aug 2009 12:12:45 +0000

Nice piece of information. Thanks for sharing.

By: Scribd での Squid によるキャッシング方法 | Carpe Diem

Scribd での Squid によるキャッシング方法 | Carpe Diem — Fri, 07 Aug 2009 06:34:28 +0000

[...] Alexey Kovyrin’s Blog に Scribd での Squid によるキャッシング方法が紹介されていました。 [...]

By: victori

victori — Thu, 06 Aug 2009 06:52:46 +0000

I concur on the Varnish statement. No matter how hard I try to get Varnish to work out for our production needs, Squid still produces better results, even if it is limited to a single core. Here are some of the issues the Varnish developers fail to mention.

1. Varnish needs to buffer the response, there is no streaming responses like Squid. This is a huge performance loss on cache misses.

2. Varnish has contention issues with small object allocations with it’s mmaped file. This creates a lot of issues when your throwing lots of small thumbnails into the cache.

3. Varnish at least in my experience has issues with keepingsockets in CLOSE_WAIT for extended periods of time. For example, Squid would only keep 430 sockets in CLOSE_WAIT while varnish would have it around keep 3,000 or so.

4. Squid has two features that make it a killer cache; stale-if-error and stale-while-revalidate. You can keep your services extremely responsive if you play your cards right.

Look at the “Shopping List” http://varnish.projects.linpro.no/wiki/PostTwoShoppingList

Search for “Streaming pass/fetch” and “Small object handling”

However, all that said I would rather ideally use Varnish than Squid. Varnish has a lot more flexibility when it comes down to HTTP acceleration. Too bad in practice Squid wins out.

Just my two cents.

By: Jim

Jim — Tue, 04 Aug 2009 20:34:24 +0000

Nice. Fast IO makes sense, though I’m puzzled that there was no reasonable RAID config faster than JBOD. Would you mind sharing a bit more around that?

Sharding the squids in a deterministic fashion via a consistent hash is also a nice touch. I’m curious how you handle failover and recovery for a given squid — I’m guessing you hash into a large number of shards, assign multiple shards to a particular squid, then dynamically balance shards across however many live squids you have at a given moment … but that’s just guessing. Could you say a bit more about that, too?

Thanks for the excellent series!

By: Scoundrel

Scoundrel — Tue, 04 Aug 2009 16:27:15 +0000

I’ve discussed this question with Percona guys and according to them they had some really weird problems with Varnish when tried to use it on one of their (pretty large) projects. So I’ve decided to go with Squid (especially after reading wikipedia and fickr guys presentations on their architecture).