MASSCAN is an incredibly fast TCP port scanner, that with the right equipment, can scan the entire Internet in under five minutes.
MASSCAN does a wonderful job at scanning the entire internet randomly, but it doesn’t cluster its results by port, which is a feature I need.
TL;DR: I wrote MASSCAN-Cluster to classify the results efficiently.
Motivation
When scanning multiple ports, MASSCAN dumps all the data into one file. It can also rotate the output file by time or size. In order to efficiently process results and extract banners, I need to classify each results by port.
The naive approach would be to read each result file, classify, and run the proper banner extraction / DPI algorithm.
My approach is to classify the results on the fly, to avoid unnecessary I/O and waste of CPU cycles.
How does it work?
Configures MASSCAN to write all its output to one output file.
less is a terminal pager program on Unix, Windows, and Unix-like systems used to view (but not change) the contents of a text file one screen at a time.
To add syntax highlighting to less on linux, first install source-highlight, then copy the following snippet to your ~/.(bash|zsh|fish|etc)rc file:
# The following adds syntax highlighting to various programming languages # Credit goes to Ter Smitten: https://goo.gl/64YU4u export LESSOPEN="| /usr/share/source-highlight/src-hilite-lesspipe.sh %s" export LESS=' -R '
# The following adds syntax highlighting to man pages # Credit goes to Todd Weed: https://goo.gl/ZSbwZI export LESS_TERMCAP_mb=$'\e[01;31m'# begin blinking export LESS_TERMCAP_md=$'\e[01;38;5;74m'# begin bold export LESS_TERMCAP_me=$'\e[0m'# end mode export LESS_TERMCAP_se=$'\e[0m'# end standout-mode export LESS_TERMCAP_so=$'\E[37;44m'# begin standout-mode - info box export LESS_TERMCAP_ue=$'\e[0m'# end underline export LESS_TERMCAP_us=$'\e[04;38;5;146m'# begin underline
tpl is small utility that transforms templates to text - a bare-bone confd alternative, that follows the unix philosophy of “Do One Thing and Do It Well”.
In other words, it just transforms template files to text, and spits the output to stdout.
It’s small, fast and is only one binary! grab it from here.
Why?
I needed a small binary that can consume dynamic data from different sources. confd is awesome, but it does much more than just transform templates. plus, many times specific filters are missing and I needed a way to add new filters easily
How?
The trivial, piping case would be:
# pipe your template $ NAME=Oded $ echo'Hello {{ "NAME" | getenv }}.' | bin/tpl # or get it from a file $ bin/tpl /path/to/template/file Hello Oded
but having a full blown templating engine at your fingertips is quite useful. what if you want to create several Dockerfile(s) for different Elasticsearch versions, and even specific plugins?
FROM elasticsearch:{{"VERSION" | getenv:"latest" }} MAINTAINER Oded [email protected] {% if "PLUGINS" | getenv:"" != "" %} # install all the plugins {% for plugin in "PLUGINS" | getenv | stringsplit:"," %} RUN usr/share/elasticsearch/bin/plugin install {{ plugin }} {% endfor %} {% endif %}
now run it:
# without any arguments $ bin/tpl /path/to/Dockerfile.tpl FROM elasticsearch:latest MAINTAINER Oded [email protected]
# with VERSION env variable $ VERSION="1.7" bin/tpl /path/to/Dockerfile.tpl FROM elasticsearch:1.7 MAINTAINER Oded [email protected]
# with the kopf and marvel plugins $ VERSION="1.7" PLUGINS="kopf,marvel" bin/tpl /path/to/Dockerfile.tpl FROM elasticsearch:1.7 MAINTAINER Oded [email protected]
# install all the plugins RUN usr/share/elasticsearch/bin/plugin install kopf RUN usr/share/elasticsearch/bin/plugin install marvel
I spend most of my day (and nights) thinking, writing & editing code. I dare to say - more than most programmers I know.
I started writing code in Pascal using Turbo Pascal, then C with Turbo C. Then I moved to writing C# with Visual Studio, and were amazed by this incredible IDE.
After a few years of writing mostly C#, I shfted. I moved to Linux, and started writing code in java and python. I was so used to IDE’s that PyCharm and IntelliJ were the only viable options.
But I wasn’t satisfied, and those IDE’s felt so heavy. Then came across Sublime Text & Atom and shifted my focus to Atom.
I’ve been a heavy Atom user for a while now. But I find myself again and again spending more of my time on the terminal. And what do I do there? I edit things. Mostly on remote machines, using vim.
My knowledge of vim is very limited, and so is my productivity. So my New Year’s resolution is to master vim!
To do so I decided to remove my beloved atom, and shift ALL my focus to vim.
Steps to Mastering Vim
Consult my gifted friend, @yosyat, which shifted to vim a few years ago.
One of the neat features python < 3.6 is missing is statement evaluation is strings. In this blog post we’ll use eval, exec and compile to implement a print function with statement evaluation.
After we’re done, we’ll be able to write the following code:
name = "oded" age = 26 print("My name is: {name}, my age is: {age} and 2*3={2*3}!") # My name is oded, my age is: 26 and 2*3=6!
TL;DR: The new print statement is 1.33x slower than the regular print when evaluating statements, and 1.09x slower for regular print operations. The whole snippet can be found here.
I just spent a few days of extensive profiling. I got to a point where I had to optimize code snippets so small, that I even looked at the performance impact of creating dictionaries.
I’m going to write a separate post about profiling, optimization and the techniques I used to get 5x performance compared to the initial version.
First, I’ll give you a sneak peek: Using the dict(…) expression is 6 times more expensive than the regular {…} call.
uberlogs does all the dumb work you have to do, to make your logs easy to parse.
Drop-in replacement for logging - fully backwards compatible with logging.
Human Readable, parsable log lines (with or without color)
Machine readable parsable log lines for your favorite data cruncher (json)
Variable formatting, so you don’t have to write .format(…) or % (…) ever again
Statement evaluation like in Ruby, Koltin, Python 3.6, etc’
Handler to violently kill the process when a log on a specific level has been written
Automatic twisted log rewriting to python.logging
you can find it on github. Feel free to open issues or submit pull requests!
Performance
I was extremely performance aware while writing this library. It does a lot of magic, and most of the time python magic is expensive. I had to use every trick I could come up with (including ahead of time compilation!) to make the performance overhead as low as I can. My work isn’t done, but I reached an acceptable overhead:
$ ITERATIONS=10000 python profile.py 1> /dev/null Profiling 10000 iterations [python 3.5.2] block took 0.666 seconds, 0.00% faster than 'std %' [std %] block took 0.655 seconds, 0.02% faster than 'std %' [std .format()] block took 0.851 seconds, 0.30% slower than 'std .format()' [uber .format()] block took 0.863 seconds, 0.01% slower than 'uber .format()' [uber complex .format()] block took 0.842 seconds, 0.28% slower than 'std .format()' [uber .format() with statement] block took 0.881 seconds, 0.05% slower than 'uber .format() with statement' [uber complex .format() with statement]
Why did I write uberlogs?
I rarely use any other logging appenders than the console one. Most of the time I send my logs to a data cruncher and write horrible regular expressions to parse the data.
To ease the pain, I started formatting my logs, so they’ll be easy to parse: arg1=arg1val; arg2=arg2val But that meant I had to write the same format everywhere, and I found myself writing long, long log lines:
performance: A class that utilizes slots gains ~30% faster attribute access. Guido wrote about that a few years ago.
Memory Consumption: A class that utilizes slots doesn’t have an underlying dict or weakref. The “problem” with dicts is that when initialized,they get a default size, which can be bigger than the amount of attributes you have. If the number of attributes is bigger than the initial dict size, the dict grows 2x, which is a huge waste of memory.
If your app performance or memory consumption is important, consider adding slots. Do make sure you benchmark your code before making such a change!
By default, instances of both old and new-style classes have a dictionary for attribute storage. This wastes space for objects having very few instance variables. The space consumption can become acute when creating large numbers of instances.
The default can be overridden by defining slots in a new-style class definition. The slots declaration takes a sequence of instance variables and reserves just enough space in each instance to hold a value for each variable. Space is saved because dict is not created for each instance.
namedtuples
namedtuples, like the name suggests - is a tuple that has a type and named attributes. tuples memory fingerprint is small and attribute access is fast. The beauty with namedtuples is that they unpack like regular tuples, but also have named attributes.
The only “downside” is that they are immutable (like regular tuples), which might pose a problem. A possible solution is to create a class with slots, or use recordclass which is a mutable namedtuples implementation.
namedtuple Returns a new tuple subclass named typename. The new subclass is used to create tuple-like objects that have fields accessible by attribute lookup as well as being indexable and iterable. Instances of the subclass also have a helpful docstring (with typename and field_names) and a helpful repr() method which lists the tuple contents in a name=value format.