I’ve planned to read 36 books in 2016 and managed to hit that number a few hours before the NY! The best of those 36 books are listed below.
Business, Management and Leadership
Considering the new role I’ve started in January 2016 (first-time CTO of a growing startup company), my reading last year was heavily geared towards business, management and leadership topics. Here are my favourite books in this category:
- “The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers” — in my opinion, a must-read book for anybody interested in starting a company or already building one. A treasure trove of great advice for startup founders on building and managing their teams.
- “Crossing the Chasm: Marketing and Selling High-Tech Products to Mainstream Customers” — the author explains why so many companies, that find an initial product-market fit, subsequently fail to grow into leaders of their respective markets and often die a slow and painful death. The concept of a chasm and, especially, the idea of the whole product were very powerful for my understanding of what I felt in many companies I worked for — mainstream customers could not use your product unless they are provided with a minimum set of components and services to solve their problem. Very important read for leaders of modern SaaS companies, especially for API/platform enterprises.
- “Turn the Ship Around!: A True Story of Turning Followers into Leaders” — an inspiring story of a navy captain transforming one of the worst-performing crews in the fleet into a perfectly functioning team by pushing control down to individual team members.
- “The Score Takes Care of Itself” — inspiring story of one of the best sport team transformations and the man behind it, legendary coach Bill Walsh.
Few more books I found very interesting:
- “The Collapse of Parenting: How We Hurt Our Kids When We Treat Them Like Grown-Ups” — maybe it is just a confirmation bias, but I absolutely loved this book. The author focuses on a few serious problems in today’s parenting and the resulting decline in the achievement and psychological health of American children. He finally managed put into words something that was bothering me for 10 years since moving to Canada. Now that I became a parent and would have to raise a child in this environment, I was glad to hear that I wasn’t crazy not to agree with the approach that is being pushed on modern parents by American society.
- “Being Mortal: Medicine and What Matters in the End” — one of my favourite authors, Atul Gawande, explores the current state of end of life care in the USA, Canada, and Western Europe. Terrifying at first, the book makes you consider your own mortality and think about the choices you are bound to make eventually for yourself and, potentially, for your close family members.
- “Sapiens: A Brief History of Humankind” — a captivating overview of our history as human species: from 70,000 years ago until the 20th century: how we evolved, how we affected other species on the planet and how did we end up where we are today. A long, but very interesting read!
- “The Road To Sparta: Retracing the Ancient Battle and Epic Run that Inspired the World’s Greatest Foot Race” — fascinating story of Dean Karnazes (one of the most famous ultra-distance runners in the world) and his exploration of the legend of the Marathon. Highly recommended to anybody interested in running.
- “Catastrophic Care: How American Health Care Killed My Father — and How We Can Fix It” — very detailed overview of what is broken in US healthcare today. Even if you don’t have anything to do with US healthcare market, the book is a great collection of stories about side-effects of what initially looked like great ideas, but ended up screwing the system even further.
I was always a huge fan of sci-fi fiction and this past year I have discovered a few real gems that ended up on my all time favourite list:
- “Remembrance of Earth’s Past (aka The Three Body Problem)” series by a Chinese author Liu Cixin — huge universe, highly-detailed and powerful characters, timeline spanning centuries — you can find all of it here. But on top of the standard components of a good space opera, there is this previously unknown to me layer of Chinese culture, language, philosophy.
This trilogy has become an instant classic for me and is in the top-10 of my all time favourites next to Asimov’s “Foundation” and Peter F. Hamilton’s “Void”.
- Everything from Niel Gainman! Up until this year when I got exposed to his writing, I’ve never realized how much pleasure one could get from reading prose. I’m not sure how he does it, but if he were to publish a book of obituaries or classifieds, I’d be willing to read that too — I enjoyed his English so much! Favourite books so far: “The Graveyard Book” and “The Ocean at the End of the Lane”.
I hope you enjoyed this overview of the best books I’ve read in 2016. Let me know you liked it!
About 6 years ago (feels like an eternity in Rails world) working at Scribd I’ve started working on porting our codebase from some old version or Rails to a slightly newer one. That’s when I realized, that there wasn’t a ruby gem to help us manage MySQL connections for our vertically sharded databases (different models on different servers). I’ve started hacking on some code to replace whatever we were using back then, finished the first version of the migration branch and then decided to open the code for other people to use. That’s how the DbCharmer ruby gem was born.
For the next few years a lot of new functionality we needed has been added to the gem, making it more complex and immensely more powerful. I’ve enjoyed working on it, developing those features, contributing to the community. But then I left Scribd, stopped being a user of DbCharmer and the situation drastically changed. For quite some time (years) I would keep fighting to make the code work with newer and newer versions of Rails, struggling to wrap my head around more and more (sometimes useless) abstractions Rails Core team decided to throw into ActiveRecord.
Finally, in the last 2 years (while trying to make DbCharmer compatible with Rails 4.0) it has become more and more apparent, that I simply do not want to do this anymore. I do not need DbCharmer to support Rails 4.0+, while it is very clear that many users need it and constant nagging in the issues and the mailing list, asking for updates generated a lot of anxiety for me, anxiety I couldn’t do much about (the worst kind). As the result, since I simply do not see any good reasons to keep fighting this uphill battle (and developing stuff like this for ActiveRecord IS a constant battle!) I officially give up.
After some long and painful consideration I’ve decided to officially suspend the project. Here is what the suspension means in this case:
- I will stop making any changes in DbCharmer code
- Pull Requests and Issues functionality on the project repository will be disabled (I will dump the issues somewhere for future reference, but no new messages could be added)
- There will be a huge message in the project README explaining that no Rails versions beyond the latest 3.2.x are supported and there are no plans to do any development to make the code work with Rails 4.0+
- Project mailing list will be disabled
- Project website will be moved to a github domain (with the same message explaining the project status)
I’m really sorry if any of the users of the project still had some hopes regarding the Rails4 branch and potential upgrade to the newer Rails versions, but 3.2.x will be the last version officially supported by the project.
Now, here are my answers to the questions that people have asked me about this decision (I’ve talked to a few of the largest users of the project already):
Are you going to kill the repo? – No, the repository (and even the rails4 branch) are going to stay intact. I’m just going to clearly mark it as inactive to make sure people do not try to use it with new Rails versions or expect the project to support those in the future.
What about Rubygems? – All rubygems versions released up until this point will stay active and accessible as long as rubygems is alive. Though no updates will be provided for any Rails versions beyond 3.2.x.
Why not crowdfund the development? – This is a really tough issue I struggled with for a long time. The problem here is that for many years now I had been very fortunate to be in a position when I’m not motivated by the money anymore. So crowdfunding the development would only increase my anxiety 10x, while not really changing the situation on the motivation side. That’s the opposite of what I need at this point.
What if you need DbCharmer functionality on your current or some next job? – This is why I’m not deleting the repo, gems, etc and calling it a suspension and not a closure 🙂 There is a chance, that one day I will end up in a situation, where I will really need all those wonderful features I’ve enjoyed with DbCharmer for years. And I’m pretty sure, that unless there will be another project available on the market, I will try to revive the project (or build something new upon the most important pieces of DbCharmer codebase). But nobody knows what will happen, so for now the project is suspended.
Can I help with the development? Maybe send in a patch? – Another very tough issue. Accepting patches still requires a lot of time and dedication to review, understand and test them. And that is not something I want to do at this point. The only real way to resume the development of the project at this point would be to transfer the ownership to somebody else. But unless someone creates a fork, shows a true dedication to the project (making sure all the incoming changes are 100% test-covered and battle-tested, etc), I’m not ready to do that. If you have some ideas on this matter, you could ping me any time.
As a leader of a technical operations team I often have to work on technical operations engineer hiring. This process involves a lot of interviews with candidates and during those interviews along with many challenging practical questions I really love to ask questions like “What are the most important resources you think an Operations Engineer should follow?”, “What books in your opinion are must-read for a techops engineer?” or “Who are your personal heroes in IT community?”. Those questions often give me a lot of information about candidates, their experience, who they are looking up to in the community, what they are interested in, and if they are actively working on improving their professional level.
Recently, one of the candidates asked me to share my lists with him and I thought this information could be valuable to other people so I have decided to share it here on my blog.
Must-Read Books List
First of all, I would like to share a list of books I believe every professional in our field should read at some point in their life. You may notice that many of these books are not too technical or are not really related to the pure systems administration part of a techops job. I still think those are very important because technical operations work on senior levels involves much more than just making sure things work as expected. A lot of it involves time management, crisis management and many other topics that are equally important for a professional in this field.
So, here is the list (with not particular ordering, grouped by topics):
Systems and Networks Administration
Technical Operations, Architecture, Scalability
Project, Release and Time Management
For more information on interesting books for technical operations engineers, you can check out the following book lists on GoodReads:
Conferences, in my opinion, are an essential part in professional development of any engineer. Here is a list of conferences that could be useful for techops engineers:
- Surge Conference – in my opinion, this is definitely one of the best conferences dedicated to building and maintaining large web architectures. If I were to choose one conference a year to go to, it would definitely be Surge. Videos from previous years are freely available online: 2010, 2011, 2012. 2013 videos should be available soon as well.
- Oreilly’s Velocity Conference – biggest and, probably, the oldest web operations and web performance event. In my opinion, recently it became too focused on web frontend performance, though it is still a really interesting event. Complete video compilations from the conference are available for sale: 2011, 2012, 2013.
- Monitorama Conference – pretty new, but already very popular conference with interesting content for everyone interested in monitoring (which most ops engineers are). Sides and videos from the first ever Monitorama conference in 2013 are available online.
- Percona Live Conference – really awesome event for anybody who has MySQL in their stack. Huge multi-track event with talks from the best and brightest people in MySQL community. Slides and keynote videos from 2013 event are available online.
- DevOps Days – small events happening all around the world and becoming more and more popular. The major topic of these conferences is the DevOps movement, related team/project management practices, etc. Videos and slides from some of the events are available online.
Even if you do not have time to watch any of those conference videos, I think every operations engineer out there would really enjoy 2011 Surge Conference closing plenary session video where Theo Schlossnagle (one of my personal heroes in IT community) described a typical debugging session many of us go through every once in a while:
Interesting Web Resources
And last, but certainly not least, I would like to share a list of web resources I like to follow to stay up to date on the most recent news and fresh ideas within the web operations community and related areas:
Leading Industry Sites and Blogs
- MySQL Performance Blog from Percona – one of the best resources on MySQL performance
- High Scalability – awesome resource with a lot of great articles on scalability, performance and design of large scale systems
- Kitchen Soap – Blog by John Alspaw (another of my personal heroes in IT field)
- DevOps Community Planet – feed/news aggregator for the DevOps community
- DevOps Community on Reddit – not too active, but still a useful resource for getting interesting news
- Agile Sysadmin – Blog of Stephen Nelson-Smith
- obfuscurity – Blog by Jason Dixon, maintainer of Graphite, author of Descartes, Tasseo and other useful tools for metrics collection and displaying
- The Agile Admin – Many interesting thoughts on agile web operations and devops
- Operation Bootstrap – Blog of Aaron Nichols talking about many different aspects of working in operations
Engineering Blogs of Large Web Companies
- Changelog – member-supported podcast on 5by5 network talking about interesting open source projects
- Food Fight – bi-weekly podcast for Chef community
- DevOps Cafe – interviews with interesting members of DevOps community
- The Ship Show – twice-monthly podcast, featuring discussion on everything from build engineering to DevOps to release management, plus interviews, new tools and techniques, and reviews
And this is it! I hope these lists would be useful for young engineers going into the technical operations and for people who already work in this space. I am going to try to regularly update this post in the future to make sure it stays relevant for a long time.
As you may have heard, last January I have joined Swiftype – an early stage startup focused on changing local site search for the better. It has been a blast for the past 8 months, we have done a lot of interesting things to make our infrastructure more stable and performant, immensely increased visibility into our performance metrics, developed a strong foundation for the future growth of the company. Now we are looking to expand our team with great developers and technical operations people to push our infrastructure and the product even further.
Since I have joined Swiftype, I have been mainly focused on improving the infrastructure through better automation and monitoring, and worked on our backend code. Now I am looking for a few good operations engineers to join my team to work on a few key projects like building a new multi-datacenter infrastructure, creating a new data storage for our documents data, improving high-availability of our core services and much more.
To help us improve our infrastructure we are looking both for senior operations engineers and for more junior techops people that we could help grow and develop within the company. Both positions could be either remote or we could assist you with relocation to San Francisco if you want to work in our office.
If you are interested, you can take a look at an old, but still pretty relevant post I wrote many years ago on what I believe an ops candidate should know. And, of course, if you have any questions regarding these positions in Swiftype, please email me at email@example.com or use any other means for contacting me and I will try to get back to you as soon as possible. If you know someone who may be a great fit for these positions, please let them know!
- Posted in: Admin-tips, Databases, Development
- Tags: cloudera, custom, hadoop, hive, jar, java, json, serde, udf
Yet another small note about Cloudera Hadoop Distribution 4.3.
This time I needed to deploy some custom JAR files to our Hive cluster so that we wouldn’t need to do “
ADD JAR” commands in every Hive job (especially useful when using HiveServer API).
Here is the process of adding a custom SerDE or a UDF jar to your Cloudera Hadoop cluster:
- First, we have built our JSON SerDe and got a
- To make this file available to Hive CLI tools, we need to copy it to
/usr/lib/hive/lib on every server in the cluster (I have prepared an rpm package to do just that).
- To make sure Hive map-reduce jobs would be able to read/write JSON tables, we needed to copy our JAR file to
/usr/lib/hadoop/lib directory on all task tracker servers in the cluster (the same rpm does that).
- And last, really important step: To make sure your TaskTracker servers know about the new jar, you need to restart your tasktracker services (we use Cloudera Manager, so that was just a few mouse clicks ;-))
And this is it for today.