понедельник, 19 сентября 2016 г.

Semi-irregular Sysadmin Ninja's Github Digest (Vol. 21)

Hello, fellow readers!
Issue 21 of "Semi-irregular Sysadmin Ninja's Github Digest" is here. The last issue was very dry, will add more of my thoughts and funny pictures. :)
Let's go!

"A reverse HTTP proxy that duplicates requests."
"You may have production servers running, but you need to upgrade to a new system. You want to run A/B test on both old and new systems to confirm the new system can handle the production load, and want to see whether the new system can run in shadow mode continuously without any issue."

"A tiny open source spacecraft project. http://kicksat.io"

WOW. Just W-O-W. Your eyes are not lying, it's open-source spacecraft. "Our goal is to dramatically lower the cost of spaceflight, making it easy enough and affordable enough for anyone to explore space. We can do this by shrinking the size and mass of the spacecraft, allowing many to be launched together."
I hope guys will be succeeded and we'll see KickSat launch soon!

"P2P web powered by torrents and blockchain."
Rejoice, my paranoid brothers and sisters! New Internet is here! Wear our foil hats on!
It's a combination of webtorrent and blockhain to make not-seizable internet!
"When you open index.html in the browser (live demo), here's what happens:
Bitcoin address 1DhDyqB4xgDWjZzfbYGeutqdqBhSF7tGt4 is searched for the latest outgoing transaction containing OP_RETURN script. Inside the script there is a torrent infohash of webpage.html. webpage.html is downloaded from torrent via webtorrent and displayed."

"IronSSH - End-to-end secure file transfer"

"While sftp and scp use ssh to keep files secure while they are being transferred over the network, once those files hit the remote server, they are no longer protected. The ironsftp executable provides additional security. When you put a file on the server using ironsftp, the file is encrypted before it is uploaded, and it stays that way on the server. When you get a file from the server, it is downloaded then decrypted. So the file remains secure until it is at the place you want to use it - on your local machine."

"QuineDB is a quine that is also a key/value store.
If your database can't print its own source code, can you really trust it?"
Very interesting and funny project! It's simple K/V storage, written in bash4, but it's also a quine!
"When you run it, the (possibly modified) source code of quinedb is printed to STDOUT, and the results of the specific command run are printed to STDERR."

"Build javascript chart dashboards without any front-end code. Uses any json endpoint. JSON config only. Ready to go."
Long time dream is fulfilled! Yes, dashboards w/o any front-end code. !https://github.com/christabor/flask_jsondash

"LogTrail is a plugin for Kibana to view, analyze, search and tail log events from multiple hosts in realtime with devops friendly interface inspired by Papertrail."
Like "tail -f", but for ELK!


"A list of companies that allow remote work and use Python."
Yep, that simple, but maybe useful.https://github.com/mariusavram91/python-remote-companies

Games on the Github 
"list of open source games and game-related projects that can be found on GitHub"

"Bringing the power of the command line to chat http://operable.io"
"Cog is an open chatops platform that gives you a secure, collaborative command line right in your chat window. It is designed to be secure, highly available, chat provider agnostic, and to be extensible using your favorite programming language."

"BORG - A terminal based search engine for bash commands"
Searching bash-related questions on Stackoverlow not leaving your terminal!

"Easily figure out which git commit caused a given stacktrace https://pypi.python.org/pypi/git-stacktrace"
A little bit naive tool which helps you to find out wich commit is responsible for specific stacktrace. Python and Java are supported.
"⚡ Deploy stuff by diff-ing the state you want against the remote server"
Interesting deploy tool. Looks nice, but IMO it better uses real configuration management tool in this case, e.g. Salt or Ansible.

And something more for ML fans:

"Neural Generation of Regular Expressions from Natural Language with Minimal Domain Knowledge"
For now, it's purely academic project - creating regexes using natural language and machine learning.
But beware - regexes it's a start, maybe sometimes computers maybe be able to write own programs.
Why they will need people then? :)

Machine Learning is simple! You can make own TF image classifier in 5 minutes!

суббота, 3 сентября 2016 г.

Semi-irregular Sysadmin Ninja's Github Digest (Vol. 20)

Hello, fellow readers!
I'm back. Now back to the news! :)

Inspired by GNU Parallel, a command-line CPU load balancer written in Rust.
Same as GNU Parallel, but modern and fast.

"rsync for cloud storage" - Google Drive, Amazon Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Cloudfiles, Google Cloud Storage, Yandex Files http://rclone.org

NASA's openmct
A web based mission control framework. https://nasa.github.io/openmct/
You will need it if your project is rocket science ;)
Use your OS X terminal shell to do awesome things. A curated list of shell commands and tools specific to OS X.

Job server in Go
Similar to Celery / Gearman but language agnostic and written in Go.

A curated list of Go patterns and idioms http://tmrts.com/go-patterns
Worth checking for all Go programmers.

Minio is an object storage server compatible with Amazon S3 and licensed under Apache 2.0 License https://minio.io
Defying Amazon's vendor lock. Didn't try it, though.

Mozilla's http-ovservatory
HTTP Observatory https://observatory.mozilla.org/
"The Mozilla HTTP Observatory is a set of tools to analyze your website and inform you if you are utilizing the many available methods to secure it."
Special bonus for machine learning lovers!

Image super-resolution through deep learning
"From left to right, the first column is the 16x16 input image, the second one is what you would get from a standard bicubic interpolation, the third is the output generated by the neural net, and on the right is the ground truth."
Looks like magic -

PArallel Distributed Deep LEarning http://www.paddlepaddle.org/
"PaddlePaddle (PArallel Distributed Deep LEarning) is an easy-to-use, efficient, flexible and scalable deep learning platform, which is originally developed by Baidu scientists and engineers for the purpose of applying deep learning to many products at Baidu."

суббота, 27 августа 2016 г.

Semi-irregular Sysadmin Ninja's Github Digest (Vol. 19)

I was slacking for a long time, I know. Sorry for that. I'll push two issues in a row now, this is the second one.

1. blessed-contrib
Build dashboards using ascii/ansi art and javascript

2. dashiell
A websocket-y frontend to osquery and facter. http://dashiell.io

3. https://telekomlabs.github.io/
T-labs, official Deutsche Telekom R&D department Github page.
Home of FirefoxOS and other projects -  worth to check.

4. jetpack
FreeBSD Jail/ZFS based implementation of the Application Container Specification

5. cachet
An open source status page system is written in PHP https://cachethq.io

6. nginx-resources
A collection of resources covering Nginx, Nginx + Lua, OpenResty and Tengine http://www.cambus.net

7. socketplane
SocketPlane - Multi-Host Container Networking

8. h2o
H2O - an optimized HTTP server with support for HTTP/1.x and HTTP/2

Semi-irregular Sysadmin Ninja's Github Digest (Vol. 18)

I was slacking for a long time, I know. Sorry for that. I'll push two issues in a row now, this is the first one. Will try to make it more regular, will include other sources too.

1. The Crystal Programming Language 
New programming language, named Crystal. "We love Ruby’s efficiency for writing code. We love C’s efficiency for running code. We want the best of both worlds." Programs look like Ruby, but compiles to efficient native code, and has compile-time error evaluation like Rust. Worth to check out, if you're PL freak, like me. :)

2. chef-koans
An experimental, test-driven way to learn about Chef.
"An experimental, test-driven way to learn about Chef. Takes some inspiration from Ruby Koans and from other things that are awesome and simple." Unfortunately, only lesson number 0 is ready now - but you're welcome to contribute, of course!
Also, if you didn't read Vim koans or Git koans - please try, it's quite fun.

3. node-bell
Real-time anomalies detection for periodic time series.

4. streem
prototype of stream based programming language
A prototype of new PL from an author of Ruby - Yukihiro "matz" Matsumoto. It's on very early stage of development.

5. sfs
Asynchronous Filesystem Replication

6. gitfs
Version controlled file system

7. awesome-courses
List of awesome university courses for learning Computer Science!

8. mochi
Dynamically typed functional programming language

9. consul-do
Do something based on leadership status

10. openbay
The Pirate Bay source code.

суббота, 21 февраля 2015 г.

Introducing collectd-iostat-python

Collectd-iostat-python is an iostat plugin for collectd that allows you to graph Linux iostat metrics in Graphite or other output formats that are supported by collectd.
This plugin (and mostly this README) is rewrite of kieran's collectd-iostat in Python and collectd-python instead of Ruby and collectd-exec

Why ?

Disk performance is quite crucial for most of modern server applications, especially databases. E.g. MySQL - check out this slides from Percona Live conference.
Although collectd provides disk statistics out of the box, graphing the metrics as shown by iostat was found to be more useful and graphic, because iostat reports usage of block devices, partitions, multipath devices and LVM volumes.
Also this plugin was rewritten in Python, because its a preferable language for siteops' tools on my current job, and choice of using collectd-python instead of collectd-exec was made for performance and stability reasons.

How ?

Collectd-iostat-python functions by calling iostat with some predefined intervals and push that data to collectd using collectd-python plugin.
Collectd can be then configured to write the collected data into many output formats that are supported by it's write plugins, such as graphite, which was the primary use case for this plugin.


Deploy the collectd python plugin into a suitable plugin directory for your collectd instance.
Configure collectd's python plugin to execute the iostat plugin using a stanza similar to the following:

Once functioning, the iostat data should then be visible via your various output plugins.
In the case of Graphite, collectd should be writing data to graphite in thehostname_domain_tld.collectd_iostat_python.DEVICE.column-name style namespaces. Symbols like '/','-' and '%' in metric names (but not in device names) automatically replacing by underscores (i.e. '_')
Please note that plugin will take only last line of iostat output, so big Count numbers also have no sense, but Count needs to be more than 1 to get actual and not historical data. And please make Interval * Count << Collectd.INTERVAL (20 seconds by default). I found e.g. Count=2 and Interval=2 works quite well for me.

Technical notes

For parsing iostat output I'm using jakamkon's python-iostat python module, but as internal part of script instead of separate module because of couple of fixes - using Kbytes instead of blocks, adding -N to iostat for LVM endpoint resolving, migration to subprocess module as replacement of deprecated popen3, objectification etc.


Maybe some data aggregation needed, e.g. we can use some max / avg aggregation of data across intervals instead of picking last line of iostat output.

вторник, 9 декабря 2014 г.

Semi-irregular Sysadmin Ninja's Github Digest (Vol. 17)

1. reptyr
Reparent a running program to a new terminal

Quite old tool made by @nelhage, it seems he is actively developing it again. It is really changes terminal for process.  "'reptyr PID' will grab the process with id PID and attach it to your current terminal. After attaching, the process will take input from and write output to the new terminal, including ^C and ^Z."
It is also quite interesting to know how it works - check this blog post if you are curious.

2. dockerana
Docker + Graphite + Graphana = Dockerana
It's exactly what it looks - Graphite + Graphana  packed in Docker container. Quite convenient.

3. seagull
Friendly Web UI to monitor docker daemon
Seagull is the best friend of docker which provides Web UI to monitor docker daemon. Demo site is down but screenshots looks nice. It seems that demo is working now.

4. Algorithms
Data Structures and Algorithms in Python
Not very exciting stuff, but might be useful. Just as it says, it is collection of data structures and algorithms in Python.

5. pg_shard
PostgreSQL extension to scale out real-time reads and writes http://citusdata.com/docs/pg-shard
Sharding helper extension for PostgreSQL. Nuff said, check docs.

6. peru
Maybe sometimes better than copy-paste.
Ah, nice tool. Another approach of eternal problems of dependencies on your repos. Like "git submodules" but easier. Works with Mercurial and SVN too, not only with git. Demo gif below:

7. awesome-public-datasets
A awesome list of (large-scale) public datasets on the Internet. (On-going collection)
List of many public (but sometimes not free) datasets on Internet, for your fun and big data projects.

8. rocket
App Container runtime
CoreOS creates own container instead of Docker. Quite controversial decision, check their blog for explanation.

9. instavpn
the most user-friendly L2TP/IPsec VPN server
Very user-friendly simple but secure VPN. Ubuntu, 512 MB RAM, curl -sS https://sockeye.cc/instavpn.sh | sudo bash, browse at http://IP-ADDRESS:8080 or use cli to setup.

10. shapeme
Evolve images using simulated annealinghttps://github.com/antirez/shapeme
Small toy from @antirez - it takes PNG and try to evolve bunch of triangles to copy it. Just for fun.

понедельник, 8 декабря 2014 г.

Semi-irregular Sysadmin Ninja's Github Digest (Vol. 16)

Ok, I'm still trying to finish with my old drafts and return to normal, weekly issues. Let's go!

1. devopsbookmarks.com
Website of devopsbookmarks.com http://www.devopsbookmarks.com

Cool new website which tries to collect all modern DevOps tools in one place (open-source and commercial too). And what is most exciting - everyone can participate through Github. :)

2. using-ngxlua-in-upyun
2014 Beijing OSC
It's also not standalone repo, but just code repo for this presentation from some Chinese conference. If you're interesting in Nginx + Lua / Openresty - check it out, quite good intro to subject. Don't afraid, it's in English - 

3. sshrc
bring your .bashrc, .vimrc, etc. with you when you ssh
If you're making some remote admin tasks on "not your" servers from time to time you're usually quite frustrated that working environment there is not like perfectly crafted precious configs. You can fix that problem with that script, but beware of big Vim plugins - they're need to be transferred to your home dir on remote host during every login.

4. tmux-resurrect
Persists tmux environment across system restarts.
Doing exactly that was promised - "saves all the little details from your tmux environment so it can be completely restored after a system restart (or when you feel like it). No configuration is required. You should feel like you never quit tmux."

5. Openstackgeek
StackGeek OpenStack Deploy
"StackGeek provides these scripts and this guide to enable you to get a working installation of OpenStack Icehouse going in about 10 minutes."
Nuff said.

6. weave
The Docker Network
Very interesting project, missing part of Docker, really. Networking is still weakest part of Docker IMO, and this project will help you with creation of virtual networks for your containers:

7. ZeroTierOne
Create flat virtual Ethernet networks of almost unlimited size. https://www.zerotier.com/
This project is similar with previous one, but main target of it is "normal" VMs and clouds and not containers. Looks quite mature and feature-full.

8. msr-cloud-tools
MSR Cloud Tools
Again, another tools from Brendan Gregg. For this time you can check is your cloud "hardware" support TurboBoost or read CPU temperature directly from CPU's MSRs (Model Specific Registers).

9. pcstat
Page Cache stat: get page cache stats for files on Linux
Yes, that tool can show for given file how many memory pages lies in Linux' file cache. Nice to know for tuning DBs, e.g. Cassandra (that's why it was written for). Not like very new tool, you can use fadvise tool from https://code.google.com/p/linux-ftools/ too - but Go code looks prettir IMO.
Also TIL mincore(2) syscall on which both tolls were based on.

10. lsleases
list assigned ip from any device in your network

Simple DHCP sniffer - will list all IP/MACs from devices in your network. Could be useful.

11. inspeqtor
Monitor your application infrastructure!

"Famous" inspector tool - modern rewrite of Monit on Go language with extended syntax and commercially available extension (because of which it was DMCAed by Monit developers first, but they're dismissed their claim after)

12. puppet-catalog-diff
Tool to diff Puppet catalogs

"A tool to compare two Puppet catalogs. While upgrading versions of Puppet or refactoring Puppet code you want to ensure that no unexpected changes will be made prior to doing the upgrade."
Very useful tool for upgrade Puppet between versions, indeed.

13. logsend
Logsend is high-performance tool for processing logs

"This like Logstash but more tiny and written by Golang. Supported outputs: influxdb, statsd and
MySQL". If you need some tool for log processing, but logstash looks somewhat bloated - check it out.