Had a red flag on my ElasticSearch cluster these days and found that the reason was related to an unassigned shards between the nodes.
As the data I collect is not that sensitive I could easy delete it and recreate in case I need in the future. But first we need to find it. There is many articles on the internet to help one to understand the shards allocation but I offer here a simple solution which is – simply delete the bastard.
First, we check on the cluster health and get the count of unassigned shards.
The concept of diceware is pretty awesome! You can read the nasty details here. It requires you to have a dice… lame! Who will bring a dice to work to generate passwords!? Come on?! The principle is really cool, regardless.
We need some code to do it. This lady did it! It is meant for you to use words that you will remember without loosing the security aspect and prevent you from using “abc123”.
What is the composition of a diceware password?
The recommended size is 4 sets of words – separated by space. The size of the words can vary from 1 to 5 characters each.
At work I wanted to make it simple and standardized, so I choose a set of 4 words with 4 letters each. You can have a list of a lot of words within that pattern. But you can use the list however you want.. The more the better. The idea is that the words are easy to remember, so it has to be within your language dictionary and composed within your pattern. In my case it is Portuguese… check this one:
volt come rena xepa giro poti roxa …
So the password would be something like – “volt come giro poti”
How strong is it?
Considering the 4 sets of 4 letters words (no pun intended) I’m using, the size of it is of 152 bits. And I’m counting the space bits as well that actually has one byte ( we have 3 spaces). It basically yields a gigantic number of possibilities.. something around the 5,708990770823839524233143877798e+45. That’s right.. 45 other digits after the last one seen.
If we count the characters only without meaning + plus space. The number of combinations would be smaller but still very big. But would be hard to remember a 4 set of random letters and we would be back to this “autg xdrv gvcn xmg”, right? That’s not a choice. What we need then is a list with words that make sense in our language. So let’s get one. You can generate yours (good idea), look on the internet, or grab from a book.
Say you have finished editing your list and ended up with 1000 words in it. That would give you a 1000 * 1000 * 1000 * 1000 = 1.000.000.000.000. Yep, that is a trillion. With your crappy list of a 1000 random 4 letter words, you would get a trillion different passwords, that would be actually easy to remember.
So basically what you have to do is process that list and spit each word randomly to compose your password. It would be rolling the dice for you.
with open('words.txt','r') as f:
mywords = [line.strip() for line in f]
print 'New Password: %s %s %s %s' %(random.choice(mywords),random.choice(mywords),random.choice(mywords),random.choice(mywords))
But this is not the diceware per se. The real diceware requires you to roll the dice 5 times to get each word. So each word of your dictionary would be assigned a number that goes from 11111 – 66666 getting you a list of 7776 unique words. Than our calculations becomes even more interesting now. Resulting in 7776 * 7776 * 7776 * 7776 = 3.656.158.440.062.976. I don’t know how to say that number in English! This is where the trues randomness in python needs to be explained, because I’m not rolling no freaking dice 5 times!
True or Pseudo Randomness
In computers system true randomness (rolling the dice) is hard to be achieved. Randomness is described as follow:
Randomness is the lack of pattern or predictability in events. A random sequence of events, symbols or steps has no order and does not follow an intelligible pattern or combination. Individual random events are by definition unpredictable, but in many cases the frequency of different outcomes over a large number of events (or “trials”) is predictable. For example, when throwing two dice, the outcome of any particular roll is unpredictable, but a sum of 7 will occur twice as often as 4. In this view, randomness is a measure of uncertainty of an outcome, rather than haphazardness, and applies to concepts of chance, probability, and information entropy.
In python for example the pseudorandomness method is used and is based on a set of mathematical functions called Mersenne Twister. In python the function “random” is used to generate a sequence of numbers and it takes a “seed” to start off. That is a deterministic way of generating numbers. You can choose that seed but generally the time of the system in milliseconds from epoch (1970) is used. Let me give you an example.
from random import seed
from random import random
# seed random number generator
# generate some random numbers
print(random(), random(), random())
# resetting the seed to 1 again
# see the pseudo thing happening
print(random(), random(), random())
You get two sets of random, but predictable numbers like the following.
You can see that after resetting the seed value, the randomness started off again from the same point, and the “randomness” is the same from the that point onwards.. hence the term pseudorandomness and deterministic.
As we set the seed number to 1, the random numbers will be given within the interval 0 and 1. Predicting the randomness can be useful to be used in production financial, engineering or machine learning systems.
If we use the python pseudo random function in a list, without setting a seed value (there is no point in it anyway) the result will be given based on a uniform likelihood or in other words, the choices are distributed evenly. In a list of 1000 words like the one I used, the likelihood of a given word to be given as a results is 1/1000 or 0,1%.
All that to say, that we don’t really need to roll the dice five times since the “entropy” is embedded in the python function.
If none of that is sufficient for you, you can order a true diceware password (made on paper) for 2 dollars.
This Feb 28th is the so called “Thesis Defense” day. It is where me, myself and I, after submitting the theses papers, put myself at the disposal of the thesis committee. In this case, “defend” does not imply that a I will have to argue aggressively about my work (although I see myself doing it).
Rather, the thesis defense is designed so that faculty members can ask questions and make sure that students actually understand their field and focus area. It serves as a formality because the paper will already have been evaluated ( have been… it is called Qualification Process). During a defense, a student will be asked questions by members of the thesis committee. Questions are usually open-ended and require that the student think critically about his or her work. The event is supposed to last from one to 3 hours, I have heard it could take more.. geees!
The Defense, is the crowning event of at least 2 years of hard study, dedication and sacrifice. And I want to tell you what I have learned with it.
It is not as hard and mystic as it seems
At least here in Brazil, masters, or as we call it “Mestrado” is not as common as it should be. It has some sort of mysticism around it, like if it was reserved for a certain “class” of student and society. People really tend to go for an MBA. MBA in Brazil, although stands for – Masters in Business Administration, has nothing to do with masters Strict Sensu and it just a Lato-Sensu course, or a specialization. I’m not taking out the credit of those that chooses the MBA, but it is different. The MBA, in the country has a more commercial practical focus. Also it is offered during the night, or weekends which helps a lot those that actually have a job to attend. I guess that is the main reason people tend to go that way. In the other hand a full blown masters course, is considered too academical or meant for those that want to pursue a professor, researcher or academic career. This is not entirely true! You can enroll to a master course, continue to work in the industry and solve a real world problem.
This is part of the mysticism that goes around a Masters course. It is not only meant for academics, it is not meant only for super nerds researchers and you can do it while you keep your actual job! It is true, that the work you have must give you certain flexibility and freedom to cope with the crazy schedules that some schools push, but it is possible.
You can have a job that is not related to universities and academic world. You can work anywhere you want. In fact, big tech companies are the ones that employs masters graduates the most. And, there is a probability that your salary increases by up to 80% if you have a master degree.
The professors and board of teachers are pretty much regular people, with experience on some topics and areas of research.. but are NOT the owners of the entire knowledge. It is pretty common that the student knows more about a given topic than the professor. He is there, to help you to adjust your thought process and writing your ideas within the accepted scientific methodology but he is not a God with omniscience. You can argue, you can defend a statement, heck, you can actually fight with your professor (although not a good idea) if you think X is equal Y.
The other aspect of the mysticism of the Master course is that you are there to learn and have classes.Myth! There is no way for the school to teach each student the specifics about his work. What you actually learn is how to organize your thought process, how to treat numbers you may collect from your research, how to use others work to build the fundamentals of your work. That is it. Don’t expect to be there and have classes of advanced math or signal processing or any in depth classes about your field. That’s not going to happen. Instead you have several classes of debate, several seminars to expose your ideas and have the other student to confront, challenge and disagree with you. You have some writing classes, you have basic statistics classes and a lot of seminars. That’s where the knowledge and ideas are born. That’s where you mature and learn how to “defend” your work and identify potential flaws in it, by discussing it with other people.
Since it depends on you, to understand, collect, test, treat and present the work.. it is not as hard as it seems. give it a try. You might surprise your self.
It is a victory in solitude
It is sad! I know. But it is the hard truth. There is a big chance that not a single soul around you will actually understand what you are doing. I mean, friends, family co-workers. None of them will, one – be interested about what you discovered, two – be willing to discuss it in detail. Nobody cares! If you are married, your wife will be interested about when you will finish it so she can have you back to regular life. Or when you will stop spending the night reading to give more attention to your kids, or – my case – When you will be finished to be able to request a raise at your current job!
There wont be any question about the inner details of your research, and if they initially show interest.. that is rapidly lost when you start going on and on about it.
Upon completion, your friends will be excited to know that you have finished it with success.. remember the myth that it is super hard and reserved for some people? Sure, they will be thrilled with the news. But don’t think they are really interested in bits and bytes of it. And it is not because the don’t like you, or have no interest on your stuff.. it is just because the don’t understand it.. and it is really hard to relate with something you have no idea about.. they are (as everybody else) afraid to look stupid.
Isn’t it sad? That you research, and successfully develop something that could be used by society and have the potential to put your name in the field history.. and nobody cares?! You can’t share it, or brag about it :)?! Come on!! So sad!
It is indeed a victory in solitude. Be glad you made it. The hours, the sacrifice is totally worth!!. Knowledge is one the few things you can actually keep and its value can’t be measured. It is however a satisfaction that only you will feel in its fullness. It is ok!
Are you curious to know what I have studied? Probably not, Maybe?!
My thesis is called – Image Perceptual Hash applied for Video Copy identification. and here is the abstract.
With the event of the Internet, video and image files are widely shared and consumed by users from all over the world. Thus, methods to identify these files have emerged as a way to preserve intelectual and commercial rights. Content based identification or perceptual hashing is the technique capable of generating a numeric identifier from the image characteristics. With this identifier, it is possible compare and decide if two images are equal, similar or different. This study has as objective discuss the application of image perceptual hashing to identify video copies. It proposes the usage of known and public methods such as the Average and Difference Hash that are based on statistics of the image also Phash and Wavelet hash that are based on the image frequency.
An identification technique was applied using a Hamming distance similarity threshold and the combination of perceptual hash algorithms for video copy identification. The method was tested by applying several attacks to the candidate video and the results were properly detailed. It is possible to use perceptual hash algorithms for video copy identification, and there are benefits when there is a combination of more than one of them filling performance gaps and vulnerabilities eliminating false positives
It is hard to believe, but it is true! I had not changed my Google password for over 5 years!! That’s embarrassing, I know.. but I can explain..(maybe not). Not, entirely satisfied with that news, I also checked in the site https://haveibeenpwned.com/ and the occurrences of my email and related passwords are exactly 14. Yes! 14 services in which my password or email have been leaked. I checked the services that have had data breaches in the past 5 years and pretty much all of them were in the list including LinkedIn, Dropbox and more. I even tried the Brazilian http://minhasenha.com.br but they don’t give you the list of services which your passwords is leaked, but instead they send you an email with the bloody leaked password! With that your only option is to change your password in all services you might have an account with!
I do use a password manager, but I have to admit that I use a single password for sites that I will most likely use only once. It is a weak password (not 123456) that is easy to remember and serves as default if I ever come back to that particular url. For services that matter I use a password generator that will spit a long, complicated impossible to remember set of 18 characters. Yet for Google, something mystic called 2FA,prevented me from changing it more frequently.
Two Factor Authentication Paradigm
2FA (2 factor authentication) – Is an authentication method that uses 2 pieces of information to allow your “entrance” to a given system. The 2FA is composed of two of these 3 things: Something you KNOW ( like your password), something you HAVE( like a key fob) and something you ARE (such as your fingerprint, or eyes). Nowadays the “something you have” that used to be a token with a screen displaying random numbers, is commonly replaced by a SMS that is sent to your phone. That is what I use for my Google account. There is a website that tracks downs services with 2FA, you will be surprised by the number of big companies that doesn’t have it yet.
It is true that a 2FA is more secure that simple password authentication but it is not massively used by people… for example, less than 10% of Google users have the feature activated. Apparently the more secure ( by consequence more steps involved) less is the interest and adoption. On the other side of the spectrum however is if after subjecting to these additional steps there is more likelihood of negligence with time. And that is where my theory comes in. I will call it, “The two factor Authentication Paradigm”. It is applied, only, to those 10% that have some form of 2FA activated.
It is basically the false sense of security that is immediately absorbed right after you elect your 2FA. You simply forget about it. Like, you are now part of a protected group of people, surrounded by a field of energy that won’t let any evil in. “I have 2FA, let me carry on with my life….this item can be checked as done, let’s move on”. I am a victim of such feeling as I just realized that it had been 5 years without touching base or even worrying about it. Is this true? Can we 100% trust 2FA, to the point to never check our passwords again? Questions like the below, didn’t even cross our my minds..
Can it be brute forced?
Can it be guessed?
Can the messages be hijacked/intercepted?
Can the software be hacked?
Social engineering hacked?
All of this questions can be answered positively, since most of the efforts employed by companies to deploy 2FA are around SMS – https://twofactorauth.org/. Advanced hacking techniques and social engineering that has been used and reported here, here, here , here, here, here , here… there is more…. here and here proves that 2FA is not something that you enable and forget about it. There are threats and one needs to be on top of it, to avoid being hacked, loosing money or have his life turned upside down.
These false notion of security is extremely dangerous as 2FA is indeed secure, but is not bullet proof. It is definitely not the silver bullet of authentication. Flaws can be found in the mobile network, public wireless network, password recovery procedures, emails, social network and as always (as the user now thinks he is safe), the social oversharing of sensitive data. For example, a hacker can guess your security question’s answers based on your twitter profile.
Among the possible actions that is to be implemented and considered to add layers of security to your digital life is:
In computing theory Collision is described as the event where two objects or records receive the same identifier or allocation in memory. The description also applies when one algorithm produces the same hash value for different entries. An algorithm considered robust must have as fundamental characteristic the “pairwise independence” , which in a simplified way means that the hash generated from two distinct entries must be different.
The hash functions do not have methods that allow the identification of the input items, or registration of the hash values generated from them. This means that although the pairwise independence feature is desirable and extremely necessary, collision is statistically possible since there are unlimited inputs and a limited (although incredibly big) number of possible outputs. The probability of collision is directly related to the composition (characters) and the size of the hash value. The greater the size of the hash value, the more difficult becomes the collision. The use of the word difficult and not impossible is adequate because the growth of computing power makes the calculations necessary to find these values more accessible. To illustrate how the collision works, consider a hash function represented by H, and any input value, for example, “Hello”, which as a result generates a hash value, 7878654, consisting of 7 (seven) characters numbers. Having the equivalent function:
H(Hello) = 7878654
The collision could happen when, for the same hash function H and a different entry, for example, “World”, the result generated is the same as that obtained for the entry “Hello” .
It is almost a year since I bought the Google Home device for my personal use. I bought the Google mini and a alternative version of the Google Home called Insignia just because it had a watch.
My kids just looooved the thing.. it was a great incentive for them to speak English and train their vocabulary.. you know ask for musics, small tasks, ask for address and such. But what I noticed was that after a while, they kept asking the same things from memory.. it wasn’t like they were actually learning anything new. And there was no way to change the language because back then it only spoke a few languages, like french and Spanish.. but no Portuguese.
But, I just noticed that after the latest update and the app revamp , Portuguese was available. It is just easier for the kids and wife to interact with the assistant now and we are really having fun!
Another upgrade is that for some reason the network driver of the Google home devices was not so good, it kept loosing the wifi signal and we often asked for something and it replied with something like “Sorry, there seems to have a problem with the network”. It was not showing up in Spotify as a possible device connection and a reboot would have to be done to have it available again.. now it is everywhere.. connection simply works!
It was around 3 years ago when I suddenly realized that a friend of mine had gone from Facebook. I don’t remember exactly why I was looking for him, but I just couldn’t find the guy. Went for looking for him in the messenger app and nothing, searched by his phone number, nothing.. it did take a while until I realized that he actually deleted his account from the platform. Why? Why anybody with sane mind would do such a thing? I mean, really?
The tool is really amazing!! It does connect you with that friend that you lost contact with ages ago. You can share your travel adventures or share not only your problems but your happiness. You can sell stuff, promote your business or simply read about people’s life. You can call people to act on a cause or promote a healthy debate about something. Don’t even get me started on theall the babies, birthdays and companies parties pictures that you can see. I can’t number the ways that you could positively use Facebook, it is just too many. And because of those reasons I thought badly of that particular friend. I considered him crazy and associated his disappearance to religion – He was actually studying and getting very involved into meditation and mind control stuff. Why would anybody not want to have this in their life? The thing is sooo addictive, and comes along with the feeling of community that makes you feel alone if you don’t participate of it, it is like you don’t have a voice or even if you don’t exist without it, isn’t t?
Fast forward to the present moment I found myself wondering if the platform is actually that good. With the recent change on Facebook algorithm the feed of stuff you end up seeing is so, so toxic! It has the goal of showing you more content from family and friends and according to Mark the focus should be people’s well being.
“You’ll see less public content like posts from businesses, brands, and media. And the public content you see more will be held to the same standard—it should encourage meaningful interactions between people.”
Come on, be honest with yourself, have you really felt good about it? The change actually made the thing to be worse. You are doomed to see the posts over and over, and doesn’t matter how many times you come back to it or device you use. It has a huge impact on marketing and ads, but also on the quality of the content you ended up seeing. Additionally there are numerous scandals that have been striking Facebook and its users with data leakage and privacy violations. Do you remember the USA elections and the avalanche of fake news? And even suspicious of Facebook selling your data to third party companies… not really a surprise, but you know… it is kinda of a bummer.
If none of that is enough reason for you to delete or stop using Facebook I can give the reason that made me think about it.
The bubble effect
The changes of the News feed algorithm favors people content like photos, events and frivolous things that it considers important. Ultimately you are trapped seeing only things that doesn’t necessarily bring any value or knowledge. The other side of it has nothing to do with the algorithm but has a lot to do with human nature where most of us try to avoid confrontation or a different opinion. Facebook really makes it easy to simply stop following that person, or block him for good. If the guy is posting to many posts related to porn, block him! If he or her has a political view that is completely different from mine.. block them! If the lady is sharing too much of her travelings and veggie recipes.. what do you do? Yes! Hit that lovely button and stop following that salad eater.
Without realizing, you are only seeing things that you like, with people that you tolerate and living in a bubble that brings no difference or diversity. The bubble is no good for anyone! It will inevitably drag you to the hate side, being it the second reason I’m getting away from Facebook.
The hate effect
Internet gives people a voice! Unfortunately the ratio of mouth and ears distribution is not that proportional. People are way more interested in saying than listening. Politics, movies, games, gay marriage, affirmative actions, immigration, guns control, abortion, adoption, divorce, human rights, minorities, civil law, criminal law…you get the idea. All topics, in my opinion, amazingly good to attract and involve people in a productive discussion where arguments are given and a healthy debate is built. But that is not what we see. Yes, internet gives you a voice but, also provides a place to hide, where you can share your stupidity and not be held responsible for it. More often than not, the so called haters can disseminate their poison and absolutely nothing happens. They get used to it.. and hate becomes their hobby.
I can’t forget one day that I had just finished the São Silvestre race and was so proud about it that I felt like sharing. I hadn’t payed the subscription and as you may suppose was not entitled to receive the medal. So, I borrowed the medal from a fellow runner and took the picture. No biggie, right? I had run the 15KM, just like anybody else.
Took exactly 2 minutes after sharing the picture to have the idiots picking on the fact that the medal wasn’t mine. 15 bloody kilometers on December 31st and so many decent things to say..and the jerks were there to judge and to spread the hate… Nope, not for me!
The envy effect
This one goes both ways. We want to check on peoples life and see what they are up to. And we also want to have people checking what we are up.. that is why we make an effort to show that our very boring festivities look like a Woodstock type of event and share the cool aspects of our life. Bottom line is, it becomes a competition to see who is the happiest, or traveled the most, or partied the most or have more success in life. We all know, no one’s life is perfect. We have down sides, we feel under the water more often than we would like it to admit. But we share only the good things…it makes me feel bad! I have the impression that the neighbor’s garden is always better than mine ( This is actually an expression in Portuguese). As people is always sharing their good moments, what was supposed to be a positive thing actually has the opposite effect. It feels like everybody is happy, super successful but me. It is freaking depressive!
One thing that must be clear as I open the door of my criticism towards Facebook is that it is a great tool. It has great qualities and I’m sure it has a place in many peoples stories of success and surely some love ones. But for me, I was simply tired of the hate, the envy and the bubble effect. I like a good conversation, I like a good debate.. even a heated but polite one. I like sharing and talking about my success. But I feel, it is not to be shared with everyone. The friendship concept promoted by Facebook is overrated, it is lazy friendship. People that actually care about you will call you in your birthdays, or will show up to help you with that home project of yours… they won’t be satisfied with a simple like in a photo that you shared, they will die to be with you during those moments. You will certainly fight with your true friends, but then you will make peace over a coffee and an awkward apology request.. or no apologies at all.. just a hug or a embarrassed look that you will understand. True friendship will cheer and be excited about your success.. gosh they will lend you money if they know it is to help you to live a dream or start a new business… or if you are just dead broke. They will be pissed at you whenever you screw up and they will tell you the truth… Facebook doesn’t give me that..
I invested many hours of my life, sharing thoughts, uploading pictures and commenting on Facebook… it is nice they provide a way for you to backup everything and download what you have there. It is quite easy to erase your account and be done with it. Not ready to delete the account entirely? I’m certainly not there yet, but I’m glad I’m not using it that much anymore… and to be free or less inclined to those feelings is just amazing!
At work we recently (not that recently) we purchased a HDMI encoder. The product comes with 16 HDMI inputs and outputs everything to the network to be consumed via RTSP, RTMP, HLS and more. However the bloody device doesn’t bring any embedded solution to merge all the outputs into a single screen. You have to treat it separately. Crazy, right?!
So, the solution then was to find something that would build the mosaic of the 16 outputs. I looked quickly for a opensource tool that would do that for you, but didn’t find anything useful. Then, had to go to old friend linux and its gizmos.
FFMPEG, I knew would do the job, but as always it would take some tinkering and lots or reading…I had no choice but go with it.. and to my surprise was quicker, simpler and funnier than I thought. Continue…