вторник, 29 июля 2014 г.

Weekly Ops Github Digest #8 (July 29th 2014)

So, it seems that our digest becoming quite regular, it's issue number 8 and number 7 was out exactly one week ago, same as issue 6 was two weeks ago, nice.

Let's try to keep it that way - new issue every Monday/Tuesday, for fresh start of new week!
... but please note that possible upcoming vacation can interfere with my intentions. :)

OK, let's go.

1. elvish
An experimental Unix shell

Nice new experimental shell (written in Go), more traditional than Xiki but also with intresting new features, like nice autocompletion and navigation mode, built in daemon for syncing command history from different instances and new Clojure/Lisp-like built-in language. Worth a try, but only if you have Linux, it doesn't compile on Mac, so, if you have one and want to be a contributor - welcome.

2. perf-tools
Performance analysis tools based on Linux perf_events (aka perf) and ftrace.

Set of performance analyzing tools for Linux, using (relatively) new Ftrace interface, written by famous Brendan D. Gregg, author of Systems Performance book, DTrace book etc.

Will not talk much about that, in short - amazing stuff, check my previous post about it.

3. Next 3 repos connected with Docker, lets merge them together:

Fast, isolated development environments using Docker http://www.fig.sh

Describe your environment in single file and run it with single command "fig up".
Looks cool? Then check out

Simple web focused Dockerfile based PaaS server.

Similar idea, but realization is little bit different - and idea is more general and limited at the same time. Just add Dockerfile to your webapp and after that run "git push" to deploy your project on choosen environment (Vagrant/Amazon/Digital Ocean/Rackspace/own server) - but only limited (but quite poular) languages and platforms are supported.

Next thing -
Service registry bridge for Docker

will help you with automatic "register/deregisters services for containers based on published ports and metadata from the container environment". I.e. you can automatically add/remove services to LB, for example. External service registers/pairing/discovery are supported also. Nice thing if your building scalable application on Docker.

Going to more general developer's stuff.

4. Two Git help pages. More general

Git Cheat Sheet

and more specific


First one is normal Git cheat sheet, but second one is more like FAQ and list of common tasks and workarounds.

5. CockroachDB.
A Scalable, Geo-Replicated, Transactional Datastore

So, it looks like NoSQL is not very popular buzzword now, NewSQL is much better!
But if talking seriously it really looks promising - it took better parts from both worlds:

ALPHA for now, we'll see how it'll going.

6. pycrastinate
TODO less, DO more. Keep your code clean without changing the way you code.

Pycrastinate is a language-agnostic tool that helps you keep your codebase (whether it is legacy or new) under control in a transparent way (i.e. without interfering with your coding). I.e. it checks for FIXMEs and TODOs and gently tring to force you to do something with that. :)
Check this talk or slides if interested.

7. Streisand.

"Streisand sets up a new server running L2TP/IPsec, OpenSSH, OpenVPN, Shadowsocks, Stunnel, and a Tor bridge. It also generates custom configuration instructions for all of these services. At the end of the run you are given an HTML file with instructions that can be shared with friends, family members, and fellow activists."
Not like very funny thing, especially for persons who struggle with internet censorshop somewhere but worth to checking then.

And time for fun!

The Permanent Web http://ipfs.io

According to website:
"IPFS is a global, versioned, peer-to-peer filesystem. It combines good ideas from Git, BitTorrent, Kademlia, SFS, and the Web. It is like a single bittorrent swarm, exchanging git objects. IPFS provides an interface as simple as the HTTP web, but with permanence built in. You can also mount the world at /ipfs."
Very experimental thing for now, but maybe it's the source of new Internet, who knows?

воскресенье, 27 июля 2014 г.

Iosnoop is a great tool, and Gregg has more!

Hello !
So, did you already missing for Github digest? OK, OK, it's in progress, gimme more time please.
But now I want to talk about new exciting tool, again.
If you consider yourself as experienced Linux sysadmin and still do not reading Brendan Gregg's blog - you are making big mistake! Please follow it through RSS or follow Gregg's twitter @brendangregg right now - I will wait. :)
Before Gregg was working in Joyent and most of his posts was connected with DTrace, but now he moved to Netflix, and most of his work now connected with Linux.
So, Gregg was ported own iosnoop tool for Linux now. What is iosnoop? It's like a tcpdump, but for disk activity. It's hard to believe, but despite of its practical top of 10000 IOPS - really iosnoop is a bash script (more precisely - combination of bash with optimized awk) and it works out of the box on most modern kernels (even debug symbols not required)!
You can say - 'Ok, it's cool new small nifty tool, which provides some Kbs of text output, meh'. Not really impressive, right? But Gregg also provide cool thing to visualize that (or similar) data - heatmaps - and also provide tooling for that. Check that great post for making latency heatmap from iosnoop output.
So, let's try that stuff on something practical. I'm suspecting that newly installed KVM server is not behaving OK, I/O seems too slow. How we can detect that? Ok, let's reboot some VM, run iosnoop for 180 sec and compare heatmaps of reboot process.
Let's do that first on some normal server first:

I cap output at 100ms, just for simplicity, but it's not critical - real value took about 150 ms, because of queueing. Server was rebooted near 80 sec timepoint, and process was finished near 120 sec.
But compare it with target server:

It looks definitely not OK. Results are much more scattered, and queueing effect looks really terrible - top delay reach almost 6000 ms! Why? It's another question, I'll try to find that tomorrow. :)
One more time - please check Gregg's blog and his github repo - you can find much more cool things there!

понедельник, 21 июля 2014 г.

Ops Github Digest #7 (July 21st 2014)

Hello Everybody!

Summer is going on, it's too hot for thinking much about work and stuff, everybody is want to go to vacation, right?
Right, but for everyone else trapped in stifling offices - next issue of Ops Github Digest!

1. Anode - Utility for analyzing graphite metrics. 
Experimental package.
Anode is a tool for experimenting with different analysis algorithms on metrics and other time series.
You have your data from Graphite, but don't know what to do with it? Try to experiment with that tool to find out:

Some math required. :)
Please check that link for explanation how it works.

2. More awesomeness!
We have another list of awesome lists - https://github.com/sindresorhus/awesome, awesome algorithms - https://github.com/tayllan/awesome-algorithms, awesome machine learning - https://github.com/josephmisiti/awesome-machine-learning, and awesome bigdata - https://github.com/onurakpolat/awesome-bigdata
But my favorite - list of awesome T-Shirt sites - https://github.com/elder-cb/teesites !

3. Log file navigator http://lnav.org

Nice log reading / colorizing tool, especially for own server.

4. Another PuppetDB frontend:
Spotify's Puppetexplorer https://github.com/spotify/puppetexplorer

Has own query language and able to combine data from different PuppetDBs into one.

5. Facette
Time series data visualization and graphing software http://facette.io/

Facette is a web application to display time series data from various sources — such as collectd or Graphite — on graphs, designed to be easy to setup and to use.

6. storm is a command line tool to manage your ssh connections. 

7. maestro-ng - Orchestration of Docker-based, multi-host environmentshttps://github.com/signalfuse/maestro-ng
Quite self-explanatory, please check if you use Docker.

8. miniLock - miniLock is a small, portable file encryption software.

New generation of modern file encryption software, which tries to proof that JS-crypto suck less nowadays.
9. And now - for you, my fellow esoteric programming language freaks!

Escher: A language for programming in metaphors 
Truly mind blowing if you have enough patience to wrap your mind around it. :)

понедельник, 14 июля 2014 г.

Sporadic Ops Github Digest #6 (July 14th 2014)

Hi All!
I'm (again, still fighting with lazyness - and overcoming it) presenting our next digest:

1. cv - Linux tool to show progress for cp, rm, dd, ...

One image is better than thousand words:

2. powerline-shell - A beautiful and useful prompt for your shell

Yes, it is nice shell from picture above:

3. awesome-perl - A curated list of awesome Perl frameworks and libraries.

At last! Perl!!!!11111

4. awesone-awesome-awesome - A a curated list of curated lists of awesome lists.

Ha-ha-ha-ha! Made my day!
And of course, there's awesome-awesome-awesome-awesome list too - for now, containing only list above...

5. More stuff for Python learning:
https://github.com/rasbt/python_reference - Useful functions, tutorials, and other Python-related things
https://github.com/s16h/py-must-watch - Must-watch videos about Python

6. The automated Pokémon speedrunning machine

An automated computer program that speedruns Pokémon, which you can run on own computer. I'm not big fan of Pokemon and speedruns, but fan of old consoles. :) Also nice piece of good Lua code with AI.

четверг, 10 июля 2014 г.

Sporadic Ops Github Digest #5 (July 10th 2014)

"Did you missed me?" :)

Was quite busy. So, let's continue our fascinating journey to amazing world of interesting and/or useful (to the Ops engineers) Github projects!

1. A shell console with GUI features http://xiki.org

Did you see that awesomeness? If not - go to http://xiki.org and give it a try. It's definitely some fresh air for old and stale world of 70's unix shells.

Did you start own "awesome" list of something? Not yet?
Now we have bunch of them - Scala, Go, but my favorite is :

2. Awesome awesomeness - A curated list of awesome awesomeness.

3. A Python library for automating interaction with websites.

Same thing as Perl's WWW-Mechanize and Ruby's Mechanize - but compatible with python 2.x and 3.x and using Requests and BeautifulSoup as engine.

4. Snabb Switch: Fast open source packet processing

Very interesting project of open-source Lua-powered bare metal ethernet switch/router/firewall. Check that article for start - Snabb Switch - Skip The OS And Get 40 Million Requests Per Second In Lua

5. The next-generation email platform 

Also very interesting project - Mail retrieval engine + API - you can collect email from various sources and retrieve it in your app/service - python-based and ready to run on your laptop or server.

6. 500 Lines or Less

Not very interesting for now, but put it to your bookmarks or watch it on Github. New "The Architecture of Open Source Applications"-like book will be there soon.

7. Easily use tmux to collaborate with distant friends!

Small tool for pair programming over sharing your terminal with tmux.

8. Flappybird clone for Vim

Just for fun! Awesome stuff! :)

суббота, 5 июля 2014 г.

Sporadic Gihub Ops Digest #4 (July 5th 2014)

I'm continuing to collect interesting Github repos (for Ops people but other geeks too):

1. Awesome Ruby.
A collection of awesome Ruby libraries, tools, frameworks and software https://github.com/markets/awesome-ruby

We have a Amazing PHP, Python, Ruby and sysadmin repos. But where's Perl ?

2. Vex.
Run a command in the named virtualenv. https://github.com/sashahart/vex

3. Rainbow Stream.
Terminal-based Twitter Client. https://github.com/DTVD/rainbowstream

4. statsrelay
A consistent-hashing relay for statsd metrics https://github.com/uber/statsrelay

5. EulerPy
Python-based Project Euler command line tool.

6. ReplayLastGoal
Automatically create an animated gif with the latest goal at the world cup

Just for fun!

среда, 2 июля 2014 г.

Sporadic Ops GitHub digest #3 (02 Jul 2014)

We are still rollin'. Go!

1. Tessera.
A dashboard front-end for graphite 
Good and promising Graphite dashboard.

2. BitAuth
Authenticate with web services utilizing the same strategy as Bitcoin.

Proposal for a new passwordless authentication protocol which is not storing user passwords at all. :)
See their blog for details.

3. cheap secrets
Store your passwords in git

Personal password storage in Git, nice. PGP encrypted, of course.

4. doitlive
doitlive is a tool for live presentations in the terminal. It reads a file of shell commands and replays the commands in a fake terminal session as you type random characters.

If you need to do presentations of your terminal sessions.

5. Ambassadord - A Docker ambassador (containerized TCP reverse proxy / forwarder) that supports static forwards, DNS-based forwards (with SRV), Consul+Etcd based forwards, or forwards based on the connecting container's intended backend (read: magic).

Too much to explain. :) Read about Docker first.

вторник, 1 июля 2014 г.

Sporadic Ops GitHub digest #2 (01 Jul 2014)

Second issue of mine Sporadic Ops GitHub digest.

1. A curated list of awesome Python frameworks, libraries and resources. 
Inspired by awesome-php

Tons of useful stuff for pythonistas.

2. A well-written BSD licensed C language TLS implementation

Another fork/rewrite of OpenSSL 'bloatware', good to know.

3. Simplistic interactive filtering tool

Cool little console tool, check screencasts for details.

4. Wrk - Modern HTTP benchmarking tool

Lua-powered multi- thread / core HTTP benchmarking tool.