Scribd is a top 100 site on the web and one of the largest sites built using Ruby on Rails. As one of the first rails sites to reach scale, we’ve built a lot of infrastructure and solved a lot of challenges to get Scribd to where it is today. We actively try to push the envelope and have contributed substantial work back to the open source community.
Scribd has an agile, startup culture and an unusually close working relationship between engineering and ops. You’ll regularly find cross-over work at Scribd, with ops people writing application-layer code and engineers figuring out operations-level problems. We think we’re able to make that work because of the uniquely talented people we have on the team.
To allow us to keep scaling, we’re now looking to add a strong, experienced operations guru to the team. As a member of Scribd operations, you’ll have tremendous ownership and responsibility for one of the web’s most popular applications. Because Scribd is a startup, you will wear many hats and have broader responsibility than you would at a larger company.
If you read this blog, you should already have a good sense of the kind of work you’ll be doing on this position.
The Ideal Profile
You are an experienced operations professional and have run ops at at least one large-scale website. You have comprehensive knowledge of a broad variety of system tools, from MySQL and Nginx to Squid and Memcached. You should also have strong software development skills and be well-versed in major programming languages. You should be strongly motivated, a creative solution finder, and ready to jump into the thorniest technical problems whenever necessary.
Responsibilities
- Develop and maintain all aspects of Scribd’s operations infrastructure, including system monitoring, backups, server configuration, databases, and caching systems
- Collaborate with engineering to create next generation infrastructure to support changing requirements
- Predict scaling problems before they occur and work with engineering to prevent them
- Write and debug application level ruby code
- Participate in an on-call rotation
- Quickly diagnose server problems and employ preventive measures to maintain high availability servers
Qualifications
- Bachelors degree in CS or equivalent experience
- 3-5 years of professional experience in site operations
- Strong software engineering skills, including knowledge of major programming languages
- Strong database skills, preferably with MySQL, and overall linux knowledge
- Experience with most of the following technologies: MySQL, Nginx, Ruby, Memcached, Squid, git, Solr, HBase, Postfix
- Proven ability to quickly learn and implement unfamiliar technologies
- Strong desire to work hard at a rapidly growing company
Location: You are preferably located near San Francisco, CA. Relocation assistance is designed on a per-case basis. In short, we’ll be creative to get you here.
Contact: Please send your email cover letter and resume with the subject “Your name – Senior Site Operations Engineer – via Kovyrin.net” to jobs@scribd.com or contact me directly using any of my contacts. All communication and correspondence is held in the strictest confidence to ensure that you can connect and learn more without exposure.
We use nginx and its features a lot in Scribd. Many times in the last year we needed some pretty interesting, but not supported feature – we wanted nginx X-Accel-Redirect functionality to work with remote URLs. Out of the box nginx supports this functionality for local URIs only. In this short post I want to explain how did we make nginx serve remote content via X-Accel-Redirect.
Read the rest of this entry »
Having a reverse-proxy web cache as one of the major infrastructure elements brings many benefits for large web applications: it reduces your application servers load, reduces average response times on your site, etc. But there is one problem every developer experiences when works with such a cache – cached content invalidation.
It is a complex problem that usually consists of two smaller ones: individual cache elements invalidation (you need to keep an eye on your data changes and invalidate cached pages when related data changes) and full cache purges (sometimes your site layout or page templates change and you need to purge all the cached pages to make sure users will get new visual elements of layout changes). In this post I’d like to look at a few techniques we use at Scribd to solve cache invalidation problems.
Read the rest of this entry »
This is going to be a really short post, but for someone it could save an hour of life.
So, you’ve nothing to do and you’ve decided to play around with IPv6 or maybe you’re happened to be an administrator of a web service that needs to support IPv6 connectivity and you need to make your nginx server work nicely with this protocol.
First thing you need to do is to enable IPv6 in nginx by recompiling it with --with-ipv6 configure option and reinstalling it. If you use some pre-built package, check if your nginx already has this key enabled by running nginx -V.
Read the rest of this entry »
After the previous post in this caching related series I’ve received many questions on hardware and software configuration of our servers so in this post I’ll describe our server’s configs and the motivation behind those configs.
Read the rest of this entry »
Month ago I was on a vacation and as usual even though our hotel provided us with an internet connection on a pretty decent speeds, I wasn’t able to work there because they’ve banned all tcp ports but some major ones (like 80, 21, etc) and I needed to be able to use ssh, mysql, IMs and other non-web software.
After a short research I’ve found a pretty simple to set up and easy to use approach to such a connection problems I’d like to describe here.
Read the rest of this entry »
Many people know me as a nginx web server evangelist. But as (IMHO) any professional I think that it is really rewarding to know as much as possible about all the tools available on the market so every time you need to make a decision on some technical issue, you’d consider all pros and cons based on my own knowledge.
This is why when I received an email from Packt company asking if I’d like to read and review their book on Lighttpd I decided to give it a shot (I usually do not review any books because I do not always have enough time to read a book thoroughly to be able to write a review). So, here are my impressions from this book.
Read the rest of this entry »
Since the day one when I joined Scribd, I was thinking about the fact that 90+% of our traffic is going to the document view pages, which is a single action in our documents controller. I was wondering how could we improve this action responsiveness and make our users happier.
Few times I was creating a git branches and hacking this action trying to implement some sort of page-level caching to make things faster. But all the time results weren’t as good as I’d like them to be. So, branches were sitting there and waiting for a better idea.
Read the rest of this entry »
Today I’ve managed to finish initial version of our bounces-handler package we use for mailing-related stuff in Scribd.
Bounces-handler package is a simple set of scripts to automatically process email bounces and ISP‘s feedback loops emails, maintain your mailing blacklists and a Rails plugin to use those blacklists in your RoR applications.
This piece of software has been developed as a part of more global work on mailing quality improvement in Scribd.com, but it was one of the most critical steps after setting up reverse DNS records, DKIM and SPF.
The package itself consists of two parts:
- Perl scripts to process incoming email:
- bounces processor — could be assigned to process all your bounce emails
- feedback loops messages processor — more specific for Scribd, but still – could be modified for your needs (will be released soon).
- Rails plugin to work with mailing blacklists
For more information, please check our README file. If you have any questions, comments or suggestions, please leave them here as a comments and I’ll try to reply as soon as possible.
Question: Do you think you have what it takes to take a service from a few hundred thousand users to tens of millions of users in 1 year flat? If you do read on and perhaps become the next beloved scalability rockstar of our age.
We are looking for a data charmer. A mysql magician. A code hack. A funny man. A mad man. A passionate man. Or perhaps a woman who does all these things and more.
Here’s what you gotta do:
- Pro-active and reactive performance analysis, monitoring and general database plumbing of all leaky issues.
- Work with others on the team to help maintain/improve and support the infrastructure for a high traffic, high growth site
- Optimize and tune the database day to day
- Algorithmic bent. Develop algos to quicken search times, response times, find shortest paths between various connections on site.
- Have solid low level networking/protocol/computer security skills
- Log everything. Usage stats, search stats, user behaviour stats. Draw conclusions. Constantly refine and tinker.
- Help with periodic large storage migrations
- Work intimately with operations, development, and strategy team to ensure smooth deployments of new iterations, high availability of database services.
- Understand capacity planning. Always thinking 10 steps ahead. (Whether it means looking at distributed systems services, cloud computing options, evaluating HA models used in other industries etc)
- Have a pulse on the state of the web, social media, social networking, different scalability architectures, benefits/negatives of each.
- Interest in high concurrency, distributed systems architectures.
- General low level hacking/scripting/optimizations in perl/python.
- Evaluate changing conditions in the archi
- Think creatively. No dogmatists.
Ideal skillset:
- BS in Comp Sci or equivalent
- 5+ years experiene with Linux/Unix systems
- 3+ years with MySQL in production environment
- Knowledge and experience with partitioned architectures and a database sharding techniques
- Capacity planning/high growth planning/emergency planning experience
- Passion, bordering on paranoia, for hunting bottlenecks, and optimizing IO operations
- Experience with MySQL replication
- Deep experience with MySQL internals
- Experience with performance analysis tools, storage engines, backup methodologies for MySQL
- Great perl/shell scripting experience
- Team player, self motivated, able to handle high stress situations while maintaining a calm disposition
- Great communication skills, attention for detail, and an interest in the business side of the equation of systems/scale planning
- Eat/sleep/breathe the web, startups, and the landscape of the social web
- Insomniac
We’re ready to offer an aggressive salary with tremendous upside by way of stock options, commensurate with your experience, your drive and your results.
Apply directly to:
net ‘dot’ startup ‘at’ googles mail service dot com
by sending us a CV/resume, and optionally, a link to your blog or Linkedin profile.