Leaky cauldron: A case of a vulnerable container

Containerised application development is a common engineering practice. However, putting your code in a container doesn’t make it secure out of the box.
Want to know more?
Let me tell you about 5 simple ways to make your container application secure. See my blog for details… ttyl

Is your container truly secure?

Containerised application development is a common engineering practice. However, putting your code in a container doesn’t make it secure out of the box. Yup! you heard it right. And believe me, you are not the only one! Count me on this list as well.

The moment of realisation dawn upon me when I had a conversation with the AppSec engineer (Aaron) about it. That’s how the conversation went (at least what I can recall).

Me: Hey Aaron, how are you doing mate!

Aaron: Good OZ, how about it.

Me: Same old same old. Hey, I would like to run a few things with you regarding the new service I am working on.

Aaron: Sure! Shoot

Me: Well! it will be quick as I have it all covered (me showing off). It is a containerised application so security wouldn't be an issue...

Aaron: Hold it right there. What makes you say that?

Me: (caught up a bit surprised) aa.. as I said it is CONTAINERISED.

Aaron: News flash OZ, putting your code in a container doesn't make it secure out of the box. At worst, it give you an illusion of security which can catch you off-guard.

Me: I have to be honest with you, I wasn't aware of it.

This conversation was a bit embracing but I have been a developer long enough to know what I don’t know and never shy to admit it. This conversation made me think about the security aspects of a container-based application. It started a journey to explore the notion of security in the context of a container based applications.

In this blog, i will share a quick summary of useful stuff that helped me create a better, more secure application.

Without further ado!

Drum rolls please!!


5 simple ways to improve your container security

1: Smaller container = Smaller attack surface

The container should only contain enough support structure (OS features, tools, libraries, running processes, etc) to run the desired application as expected. Everything else should not be part of your container ecosystem.

Given the hereditary nature of the container framework like Docker. You are not only inheriting the functionality but also the security vulnerabilities. It is important to evaluate the base image carefully.

One possible solution is to use Docker Slim which minify the image and create security profiles (more on it later). Check out this short video for details

Don’t take my word for it… see it in action!

This principle will guide you to adopt micro service based architecture. A smaller size image is easier to sync with remote repositories and improves the deployment time.

2: Contain the container

Control what the container can see and do!

Operating System (OS) provides a large number of system calls to perform various OS / Kernel related operations like modifying I/O privileges or profiling etc.

It is important to control the sys commands a container can run. By default, Docker applies a “Seccomp” profile which whitelists a small subset of commands. However, you can take a step further to create a tighter list.

Ensure that the container is running in the least possible privilege settings. Aside from Seccomp, you can isolate containers (limit what container can see or do) by running it in the context of an unprivileged user. DON’T USE ROOT USER unless there is a clear case for it. Remember: root user in a container is a root user outside.

3: Run vulnerability scans

Containers are created hierarchically. Your container uses a base image that may inherit from any other and so on. This establishes a system largely based on vendors or third-party routines. With this model, you will inherit the good, the bad and the ugly.

If your base image has a security vulnerability, guess what!! it has become your application’s security vulnerability. This serious problem leads to the rise of security vulnerability tools like Twist Lock, Anchore, Clair, etc. There are a lot of options out there…. open source, proprietary. In fact, Docker Enterprise also provides scan features when you push your docker image to the registry. Personally, I am using Twist-Lock and it works like a charm.

To put it simply the main purpose of the security vulnerability tool is to perform a static analysis of the dependency tree and identify any potential security loopholes by comparing it with the database of known security issues.

The scan results may look like this:

Recently, I used these tools extensively with the service I am working on. I will share my insights about comparative analysis of JRuby image on Open JDK, AWS Corretto, Adopt JDK and Azul Zulu. Let’s keep this topic for another blog 🙂

Pro tip: Integrate Security Vulnerability scan in your CI/CD pipeline and make it a blocker for your deployment.

4: Apply security best practices

I got to have this one on my list. It is always handy to have a tool for course correction, especially for a lazy lousy developer like me.

Lads! please welcome “Docker Bench Security“. Shout out to Thomas & Diogo for this Swiss army knife of a tool for docker developers.

When you run this tool on your container or image, it executes a script to check common best practices (like the stuff we discussed in point #2) and give you a quick summary and report. Running this tool while developing apps will ensure that all bases are covered.

# clone the repo and run it like this.
sh docker-bench-security.sh -i <Container or Image Name>

One area of improvement for this script. I would love to have something like this as part of my continuous development process or as a plugin to my IDE. Like to know your thoughts. If I get 10 or more yeas, I will work on it.

5: Use signed images

This may sound obvious and common knowledge but more often than not, it slips through the cracks. You can blame the triviality of the task but it can be consequential.

As a general rule of thumb, you should know the owner of the docker image you are using (as a base or an element of a composite). This is where “signed images” come into the picture. It revolves around the idea of using Digital Signature to ensure integrity.

The author should sign the images and the consumer should use one. Let’s break this down a bit.

Author Checklist

One of my signed images #self marketing

– Generate “root key” using docker trust. See the link below for more details

– Use the root key to create a repository tag pair using the docker trust signer. The public key will be shared by Docker Notary service.

– Use the private key to sign the tag.

– No need to signed all the tags. However, you should sign the ones your consumers will use. For example, latest or LTE tag, etc.

– Make this part of your build process to avoid manual work.

– Think about whether it makes sense for you to make it an Official Docker container. It is not straight forward, you need to comply with the list of requirements that may affect your timelines.

More details here.

Consumer Checklist

– Use official or user signed images.

– Enforce this check on the docker daemon to avoid non-compliance.

Application security is a vast and evolving field. I am not an expert in it by any means. The main purpose of this blog is to share my experience and things I learned by reviewing articles, documentation and going through many iterations of trial and error.

I hope it helps in your journey! Let me know your feedback.

Stay Safe! Stay Hopeful!

My Experience @ VoxxedDays 2019

VoxxedDays only started organising conferences and tech community events in Singapore recently. However, they are establishing themselves as one of the legit places for developer, entrepreneurs and techies.

My relationship with VoxxedDays goes back to 2017 when they did their first conference in Singapore and I was lucky to participate init.

This year’s conf lives up to its reputation.

Fan moment #1 “With Compiler Wizard Chris Thalinger”
Fan moment #2 “Jenkins Creator Kohsuke Kawaguchi”

Key takeaways are the sessions related to Graal (which includes mine as well), Kafka and Kubernetes. Since its my blog, i will share my session link 🙂

I also like to give a special mention to the day 1 closing note from Tim Berglund.

Checkpoint 2019

Its that time of the year when we look at the passage of day and night with more excitement.

It’s not just another day end…

it’s not just another month end…

ITS is the end of the whole freak’in YEAR

and I don’t see any reason not to celebrate it… we made it through.

Just like with everything these days, new year’s eve attracts a share crowed of negativity who argues the rationale of celebrating the new year.

I hear your point! I do!

Now, let me if new year’s celebration is not worth celebrating then what else is?

Human celebrations (almost all of them) is associated with the start of OR end of something … Independence Day, Spring Coming, Harvesting, Birthday and so on!

So, on this special occasion try not to be a drag and have some fun!

Happy Holidays, Merry Christmas and Happy New Year to everyone!

What’s up with all the silence, OZ

Wasn’t blogging for a while and let me tell you why!

Let me be the first to admit that I had been awfully quiet on my blog for a few months. All the silence of the blog is not telling the whole story of my past months. Looking back, they had been hectic. Let’s break it down a bit!

Hello Fatherhood

Became father of a baby gal (wooohoooo)… I was shit scared but holding your child is the best feeling in the whole… words can’t explain and words are not enough.

47f6d7a8-05ab-4667-aaa0-0bc85fb72fba

I wanna be a youtube star 😀

Started youtube series about serverless computing model. Episode 1 already up!!

OZvsServerless EP1

FOSSASIA 2018

Attended + Presented at FOSSASIA 2018. It was fun reconnecting with old friends and visitors from all over the world.

26097971647_aa3e4fe206_z

See the full video here: The myths and realities of serverless architecture – Owais Zahid – FOSSASIA 2018

Hello Toronto

Took a 23-hour flight to Toronto and attended Scrum Gathering. It was an amazing experience visiting this lovely city and meeting the most polite ppl on the planet :D. More on this later.

IMG_2018

Few Other things!!

Apart from the big apples! Participated in Google Code Jame, Hackathon and learned how to change diapers:D, had an argument with Lufthansa Flight (see twitter for the whole story :p)

There you have it! A quick summary of what happened. Now the dust is settled, I will add more blogs and videos (so stay tuned!)

Connecting people in Style … CSS

Why Tech communities are important? Is your organization socially connected?

Quick Question, “Do you know why Open Source / Inner Source projects are more successful and evolve at a higher rate?”

Well! there are many reasons for it, but none more important or crucial as “Community”.

The community is the set of people which have common understanding, aspirations, and concerns. From social work to social awareness, communities have been integral to the development, improvement, and sustenance of human societies.

Using the same definition, we can infer by virtue of “what’s true in the real world is true in the virtual world”. Communities or to be specific Tech Communities are integral to development, improvement, and sustenance of projects. The main reason we use a particular tool or technology has a lot to do with the number of people backing it. We frequently come across the statements which mention “number of stars” or “forks” on GITHUB, amount of documentation, tutorials available etc. All these factors are crucial to the success of a project.

Long story short, “Tech communities are important”

More so, there is a social aspect of the communities. When people are part of the group, they start to build relationships and bonds. This process not only strengthens the fabric of the unit but also accelerate learning and exchange of ideas. The idea to bring people of a community always results in innovation, creativity … spark

Continuing the traditions of social connectivity, on 29th Nov Autodesk Singapore hosted Singapore CSS meetup group. The idea is to bring Web experts and enthusiasts on a single platform and let the 2-way communication kicks in. From financial institutes to design studios, from college students to tech company; we got a crowd of over 40 people from all walks of life.

We also got an opportunity to showcase some of the awesome stuff we are doing at Autodesk. We propagate the message that Autodesk is one of the great places to work for and we value innovation and creativity. All in all, I had a great time meeting new and old friends and learning new things.

To know more about the event, see the official page. Following are some of the pictures from the event 🙂

Everyday Randomness: Guinea pig wheel

Are you going places when you move?

Chapter 2: “Guinea Pig Wheel”

Where are we going?
Nowhere it seems!
We are running in circles
Coming back to where we started
Over and over again

Are you walking when you walk? Are you running when you run?

The answer is: It depends.

It depends on your perspective. If we relate walking or any form of movement with “going places” then definitely NOT! … So… what are we doing on the treadmill?

How many of us are guinea pigging ourselves every day! This question comes to my mind when I went to my friend’s place and looked at his pet guinea pig (Sammy) Ferris wheeling the hell out of himself. Looking from outside, this whole process looks darn stupid and depressing. It brings me back to my original point of repetition in life (see my earlier blog for details).

Anyways, back to Sammy. The poor guy doesn’t have much choice; does he. But what about us? Why are we doing the same thing over and over again without going places? That visit made me think. After few days, when I start running on my treadmill, it just wasn’t the same. I didn’t feel like doing it. So, I stopped the treadmill and gave an outdoor a try.

I saw people; young and old. Trees dancing at the tone of breeze. Birds doing the last rendezvous before nightfall. I had a really good time. I was moving and at the same time going places. It is neither a rebellious deed nor a novel idea, but it did give me a glimpse of possibilities awaits us. All it takes is a little thought and effort to do things differently.

Wrap up

There you have it. This was the story of one of many Ferris wheels of my life. Do you have any Ferris wheels of your own? how are you coping with them? and most importantly, what are you doing to break them? 

~Owais Zahid

Everyday Randomness: Tai Seng Pavement

Intricacies in insignificance. How to draw wisdom from every day experiences.

Hello Readers,

Life is continuous and repetitive. It is continuous because of its very nature that can be judged on the scale of time. At the same time, it is repetitive because of our persistence with following routines and orders.

Put it this way; we are growing old every day and while growing old we are repeating things (going to work, school etc).

In our pursuit of repetition, we tend to ignore the details of our surroundings or scenarios, we ignore the experiences, the randomness that life itself throws at us.

In this blog series, I will share some of the oddities I noticed while looking at things from different perspectives.

Chapter 1: “Tai Seng Pavement”

Tai Seng MRT (train station) is the closest station from my house. I use this station for work and other errands daily. Situated in the north east of Singapore, It is a commercial and industrial area (kinda’). Its station gets crowded in the morning and evening hours.

9:05 AM. It was a day like any other. I crossed the traffic signal (my house is on the other side of the station) and stepped on the pavement (sidewalk) which led to the station. It was a 20 – 25-meter strip and passing it was a matter of few seconds.

Sound simple! I am sure it does. However, there is one small problem.

In spite of the ample side space, the concrete pavement is only 1 meter wide. You don’t want to walk on the grass when it rains (and it rains all the time). Secondly, as I mentioned Tai Seng is a commercial/industrial area, so a lot of people come out the station when I am trying to get in and vice versa in the evening.

In short, I was on a narrow pathway with people charging towards me (i am being dramatic here :D). At that moment, it occurred to me that this seemingly random encounter with a herd of people (mulling me) is more than just a random event. It was thought-provoking and has the wisdom to share.

Few things crossed my mind while I took a moment and reflected on my daily struggles to overcame 25 meters passage. Let me share some of them with you.

  • Going against the flow. It demands perseverance and patience.

  • Plan to address problems that are recurring in nature. Like stuck in a traffic, health problems etc.

  • The action plan you choose to address your everyday recurring problems tells a lot about what kind of a person you are.

Let me elaborate the last point. In fact, let me explain this in the context of “tai-seng pavement”. There can be three possible solutions to avoid being stampeded (again being over dramatic…. more than three solutions are also possible… use your imagination :))

Action Plan 1: All’in

This is the most common and most convenient (no brainer) option. Ride your luck and see where it takes you. This is my goto option and it works most of the time (minus the pushing and shoving). In fact, this is the go-to option for all of us, if the problem in-hand is not significant.

Tips: Open your favorite app and keep walking while staring at your cell phone. If you are bulky like me, then it is a bonus 😀 

Action Plan 2: Grease Monkey

Be attentive and figure out how can you safely negotiate the passage. This action plan is important in tough conditions, like thunderstorms, people with umbrellas or you have grocery bags etc.

In everyday life, we use this plan when a seemingly simple scenario becomes difficult because of variance. For example, using a sharp knife to cut meat or cycling in rain.

Action Plan 3: Heisenberg Solution

Identify the problem and create a support group. Convince authorities to increase the size of the pavement. Another solution is to educate people to use on side of the pavement to go in one direction.

This action plan is for chosen few! I am not saying that majority of us are dumb; on the contrary, most of us will not give an iota of thought to solve a trivial or problems beyond our comprehension. People who use this action plans are thinkers, innovators, and inventors.

Wrap up

Enough said. Time to wrap it up. But before I end this post, I like to leave you guys with one thought. “Next time you are doing something, take a moment and look at what, why and how of the things you are doing”. I bet you will make few interesting observations.

~Owais Zahid

P.S: The title image is taken from Hayley Leibson‘s article.